High resolution time-of-flight depth imaging

ABSTRACT

Techniques and systems are provided for high resolution time-of-flight (ToF) depth imaging. In some examples, an apparatus includes a projection system including one or more light-emitting devices, each light-emitting device being configured to illuminate at least one portion of an entire field-of-view (FOV) of the projection system. The entire FOV includes a plurality of FOV portions. The apparatus also includes a receiving system including a sensor configured to sequentially capture a plurality of images based on a plurality of illumination reflections corresponding to light emitted by the one or more light-emitting devices. Each image of the plurality of images corresponds to one of the plurality of FOV portions. An image resolution associated with each image corresponds to a full resolution of the sensor. The apparatus further includes a processor configured to generate, using the plurality of images, an increased resolution depth map associated with the entire FOV.

FIELD

The present disclosure generally relates to time-of-flight (ToF)sensing. In some examples, aspects of the present disclosure are relatedto increasing the image resolution of depth maps generated based on ToFdata by sequentially scanning portions of a field-of-view (FOV) of animaging system.

BACKGROUND

Image sensors are commonly integrated into a wide array of electronicdevices such as cameras, mobile phones, autonomous systems (e.g.,autonomous drones, cars, robots, etc.), smart wearables, extendedreality (e.g., augmented reality, virtual reality, mixed reality)devices, and many other devices. The image sensors allow users tocapture video and images from any electronic device equipped with animage sensor. The video and images can be captured for recreational use,professional photography, surveillance, and automation, among otherapplications. The video and images captured by image sensors can bemanipulated in various ways to increase the quality of the video orimages and create certain artistic effects.

In some cases, light signals and image data captured by an image sensorcan be analyzed to identify certain characteristics about the image dataand/or the scene captured by the image data, which can then be used tomodify the captured image data or perform various tasks. For example,light signals and/or image data can be analyzed to estimate a distanceof the scene captured by the image data. Estimating distance informationcan be useful for a variety of applications, such as three-dimensional(3D) photography, extended reality experiences, object scanning,autonomous vehicle operation, Earth topography measurements, computervision systems, facial recognition systems, robotics, gaming, andcreating various artistic effects, such as blurring and bokeh effects(e.g., out-of-focus effects). However, estimating distance informationwith sufficient resolution and/or accuracy can be prohibitively powerand compute intensive.

SUMMARY

Systems and techniques are described herein that can be implemented toperform high resolution time-of-flight (ToF) depth imaging. According toat least one example, apparatuses are provided for high resolution ToFdepth imaging. An example apparatus can include a projection systemincluding one or more light-emitting devices, each light-emitting deviceof the one or more light-emitting devices being configured to illuminateat least one portion of an entire field-of-view (FOV) of the projectionsystem. The entire FOV can include a plurality of FOV portions. Theapparatus can also include a receiving system including a sensorconfigured to sequentially capture a plurality of images based on aplurality of illumination reflections corresponding to light emitted bythe one or more light-emitting devices. Each image of the plurality ofimages corresponds to one of the plurality of FOV portions. Further, animage resolution associated with each image of the plurality of imagescorresponds to a full resolution of the sensor. The apparatus can alsoinclude a memory (or multiple memories) and a processor or multipleprocessors (e.g., implemented in circuitry) coupled to the memory (ormemories). The processor (or processors) can be configured to generate,using the plurality of images, an increased resolution depth mapassociated with the entire FOV.

In another example, an apparatus can include means for illuminating aplurality of field-of-view (FOV) portions of an entire FOV of theprojection system, wherein the means for illuminating is configured tosequentially illuminate at least one portion of the entire FOV; meansfor sequentially capturing a plurality of images based on a plurality ofillumination reflections corresponding to light emitted by the means forilluminating, wherein each image of the plurality of images correspondsto one of the plurality of FOV portions, and wherein an image resolutionassociated with each image of the plurality of images corresponds to afull resolution of the means for receiving; and means for generating,using the plurality of images, an increased resolution depth mapassociated with the entire FOV.

In another example, methods for high resolution ToF depth imaging areprovided. An example method can include illuminating, using one or morelight-emitting devices of a projection system, a plurality of FOVportions of an entire FOV of the projection system, wherein eachlight-emitting device of the one or more light-emitting devices isconfigured to illuminate at least one portion of the entire FOV. Themethod can also include sequentially capturing, by a sensor of areceiving system, a plurality of images based on a plurality ofillumination reflections corresponding to light emitted by the one ormore light-emitting devices. Each image of the plurality of imagescorresponds to one of the plurality of FOV portions. Further, an imageresolution associated with each image of the plurality of imagescorresponds to a full resolution of the sensor. The method can alsoinclude generating, using the plurality of images, an increasedresolution depth map associated with the entire FOV.

In another example, non-transitory computer-readable media are providedfor high resolution ToF depth imaging. An example non-transitorycomputer-readable medium can store instructions that, when executed byone or more processors, cause the one or more processors to: illuminate,using one or more light-emitting devices of a projection system, aplurality of field-of-view (FOV) portions of an entire FOV of theprojection system, wherein each light-emitting device of the one or morelight-emitting devices is configured to illuminate at least one portionof the entire FOV; sequentially capture, using a sensor of a receivingsystem, a plurality of images based on a plurality of illuminationreflections corresponding to light emitted by the one or morelight-emitting devices, wherein each image of the plurality of imagescorresponds to one of the plurality of FOV portions, and wherein animage resolution associated with each image of the plurality of imagescorresponds to a full resolution of the sensor; and generate, using theplurality of images, an increased resolution depth map associated withthe entire FOV.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above can include: illuminating each of the plurality of FOVportions in a sequential illumination order, the sequential illuminationorder including illuminating a single FOV portion at a time; receivingeach illumination reflection of the plurality of illuminationreflections in a sequential receiving order that corresponds to thesequential illumination order; and generating each image of theplurality of images based on each illumination reflection of theplurality of illumination reflections. In some examples, illuminatingeach of the plurality of FOV portions in the sequential illuminationorder can include: illuminating a first FOV portion of the plurality ofFOV portions; receiving a first illumination reflection corresponding tothe first FOV portion; after receiving the first illuminationreflection, illuminating a second FOV portion of the plurality of FOVportions; and receiving a second illumination reflection correspondingto the second FOV portion.

In some aspects, the one or more light-emitting devices can include aplurality of light-emitting devices. In some cases, the projectionsystem can include a plurality of projection lenses, each projectionlens of the plurality of projection lenses being configured to projectlight emitted by one of the plurality of light-emitting devices towardsa different FOV portion of the plurality of FOV portions. In some cases,each projection lens of the plurality of projection lenses can bepositioned above the plurality of light-emitting devices. In someexamples, the projection system includes one or more diffuserspositioned relative to the plurality of projection lenses. For instance,the one or more diffusers can be configured to diffuse the light emittedby the plurality of light-emitting devices. In some examples, theprojection system includes a segmented prism array. For instance, thesegmented prism array can be configured to direct the light to each FOVportion of the plurality of FOV portions.

In some aspects, the one or more light-emitting devices can include asingle light-emitting device. In some cases, the projection system caninclude a scanning mirror configured to project light emitted by thesingle light-emitting device towards different FOV portions of theplurality of FOV portions when oriented at different orientations. Inone example, the scanning mirror can be a micro electro mechanicalsystem (MEMS) mirror. In some aspects, the scanning mirror can bepositioned above the single light-emitting device. In some cases, eachof the different orientations of the scanning mirror can correspond to adifferent orientation angle between the scanning mirror and a plane ofthe single light-emitting device. In some examples, the projectionsystem includes one or more diffusers positioned relative to thescanning mirror. For instance, the one or more diffusers can beconfigured to diffuse the light emitted by the single light-emittingdevice.

In some aspects, the receiving system includes an array of image lenses.Each image lens of the array of image lenses can be configured toproject, to the sensor, light associated with a different portion of thescene corresponding to a respective FOV portion of the plurality of FOVportions.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above can include a filter positioned above the sensor. Thefilter is configured to transmit light with a frequency corresponding toa frequency of light emitted by the one or more light-emitting devices.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above can include synchronizing the projection system and thereceiving system based at least in part on: sending, to the projectionsystem, a first control signal directing the projection system toilluminate a particular FOV portion of the plurality of FOV portions;and sending, to the receiving system, a second control signal directingthe receiving system to associate an illumination reflection received bythe sensor with the particular FOV portion, wherein the first controlsignal and the second control signal are time-synchronized.

In some aspects, a first FOV portion of the plurality of FOV portionscan partially overlap at least a second FOV portion of the plurality ofFOV portions.

In some aspects, generating the increased resolution depth mapassociated with the entire FOV can include: generating a plurality ofpartial distance measurements, each of the plurality of partial distancemeasurements corresponding to one of the plurality of illuminationreflections; and combining the plurality of partial distancemeasurements. In some examples, an image resolution of the increasedresolution depth map can correspond to a maximum resolution of thesensor multiplied by a number of the plurality of FOV portions.

In some aspects, each apparatus described above is or includes a camera,a mobile device (e.g., a mobile telephone or so-called “smart phone” orother mobile device), a smart wearable device, an extended realitydevice (e.g., a virtual reality (VR) device, an augmented reality (AR)device, or a mixed reality (MR) device), a personal computer, a laptopcomputer, a server computer, an autonomous vehicle, or other device. Insome aspects, the apparatus includes a camera or multiple cameras forcapturing one or more videos and/or images. In some aspects, theapparatus further includes a display for displaying one or more videosand/or images. In some aspects, the apparatuses described above caninclude one or more sensors.

This summary is not intended to identify key or essential features ofthe claimed subject matter, nor is it intended to be used in isolationto determine the scope of the claimed subject matter. The subject mattershould be understood by reference to appropriate portions of the entirespecification of this patent, any or all drawings, and each claim.

The foregoing, together with other features and examples, will becomemore apparent upon referring to the following specification, claims, andaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative examples of the present application are described in detailbelow with reference to the following figures:

FIG. 1 is a simplified block diagram illustrating an example imageprocessing system for time-of-flight (ToF) signal processing, inaccordance with some examples of the present disclosure;

FIG. 2A is a simplified block diagram illustrating an example of adirect ToF sensing procedure, in accordance with some examples of thepresent disclosure;

FIG. 2B is a simplified block diagram illustrating an example of anindirect ToF sensing procedure, in accordance with some examples of thepresent disclosure;

FIG. 3 is a diagram illustrating an example technique for measuring thephase angle of a light signal, in accordance with some examples of thepresent disclosure;

FIG. 4 is a diagram illustrating a continuous wave method for ToFsensing, in accordance with some examples of the present disclosure;

FIG. 5 is a simplified block diagram illustrating an example imageprocessing system for ToF signal processing, in accordance with someexamples of the present disclosure;

FIG. 6A is an illustration of an example field-of-view (FOV) of an imageprocessing system for ToF signal processing, in accordance with someexamples of the present disclosure;

FIG. 6B is a diagram of example portions of an FOV of an imageprocessing system, in accordance with some examples of the presentdisclosure;

FIG. 7 is a diagram of an example device for illuminating portions of anFOV of an image processing system, in accordance with some examples ofthe present disclosure;

FIG. 8 is a diagram of another example device for illuminating portionsof an FOV of an image processing system, in accordance with someexamples of the present disclosure;

FIG. 9A is a diagram of an example device for receiving illuminationreflections of portions of an FOV of an image processing system, inaccordance with some examples of the present disclosure;

FIG. 9B and FIG. 9C are diagrams illustrating example sequential ToFimage capturing processes, in accordance with some examples of thepresent disclosure;

FIG. 10A and FIG. 10B are diagrams of example systems for synchronizedToF sensing, in accordance with some examples of the present disclosure;

FIG. 11 is a flow diagram illustrating an example of a process for highresolution ToF depth imaging; in accordance with some examples; and

FIG. 12 is a diagram illustrating an example of a system forimplementing certain aspects described herein.

DETAILED DESCRIPTION

Certain aspects and examples of this disclosure are provided below. Someof these aspects and examples may be applied independently and some ofthem may be applied in combination as would be apparent to those ofskill in the art. In the following description, for the purposes ofexplanation, specific details are set forth in order to provide athorough understanding of subject matter of the application. However, itwill be apparent that various examples may be practiced without thesespecific details. The figures and description are not intended to berestrictive.

The ensuing description provides illustrative examples only, and is notintended to limit the scope, applicability, or configuration of thedisclosure. Rather, the ensuing description will provide those skilledin the art with an enabling description for implementing theillustrative examples. It should be understood that various changes maybe made in the function and arrangement of elements without departingfrom the spirit and scope of the application as set forth in theappended claims.

Image sensors are commonly integrated into a wide variety of electronicdevices, such as cameras, mobile phones, autonomous systems (e.g.,autonomous drones, cars, robots, etc.), extended reality (e.g.,augmented reality, virtual reality, mixed reality) devices,Internet-of-Things (IoT) devices, smart wearables, and many otherdevices. Video and image recording capabilities have become morewidespread as increasingly more electronic devices are equipped withsuch image sensors. In addition, the image processing capabilities ofelectronic devices have continuously improved, allowing image data to bemanipulated to produce higher quality videos and images, generate a widearray of artistic effects, and implement image data in a variety ofapplications. For example, light signals and image data captured bysensors can be analyzed to identify certain characteristics about theimage data and/or the scene captured by the image data. This informationcan then be used to modify the captured image data or perform varioustasks. For example, light signals and/or image data can be analyzed toestimate a distance of objects within the scene captured by the imagedata.

Time-of-flight (ToF) sensor systems can use light (e.g., infrared (IR)light, near-infrared (NIR) light, and/or other light) to determine depthand/or distance information about a target (e.g., a surrounding/nearbyscene, one or more surrounding/nearby objects, etc.). A ToF sensorsystem can include a light emitter configured to emit a light signaltowards a target, which can hit the target and reflect back to the ToFsensor systems. A ToF sensor system can also include a sensor configuredto detect and/or measure the returned/reflected light, which can then beused to determine depth and/or distance information for the target. Thedistance of the target relative to a ToF sensor system can be used toperform depth mapping. The distance of the target can be calculatedthrough direct ToF or indirect ToF.

The resolution (e.g., image resolution) of some ToF sensor systems cancorrespond to the number of reflected light signals the ToF sensorsystem can receive and process at a time. For example, the imageresolution of a ToF sensor system can correspond to the number of pixels(e.g., photosensitive elements) of a ToF sensor. In many cases, ToFsensor technologies utilize complex hardware and/or extensive processingpower. As a result, it can be impractical and/or impossible (e.g.,cost-prohibitive) to increase the resolution (e.g., number of pixels) ofa ToF sensor. For instance, some existing ToF systems are limited to aresolution of no more than 320×240 pixels. Such a low resolution may beinsufficient for various types of imaging and sensing applications, suchas long distance imaging, wide field-of-view (FOV) imaging, autonomousvehicle operation, machine vision systems, facial recognition systems,digital measurement systems, gaming, robotics, among others. Thus, it isbeneficial to develop improved ToF systems and techniques forefficiently capturing high-resolution image depth information.

The present disclosure describes systems, apparatuses, methods, andcomputer-readable media (collectively referred to as “systems andtechniques”) for providing high resolution ToF depth imaging. Thesystems and techniques described herein provide the ability for a ToFsystem to generate depth maps (e.g. depth image maps) with a resolutionthat exceeds the full (e.g., maximum) resolution of a ToF sensor of theToF system. In addition, the ToF system can generate high-resolutiondepth maps without incurring substantial delays in processing time. Togenerate high-resolution depth maps, the ToF system can capture depthinformation associated with portions of an entire field-of-view (FOV) ofthe ToF system. In some cases, the ToF system can sequentially capturedepth information associated with individual portions (referred to asFOV portions) of the entire FOV. For example, the ToF system can “scan”through the FOV portions. The ToF system can utilize a single ToF sensor(e.g., the same ToF sensor) to scan each FOV portion. The individual FOVportions can correspond to (or approximately correspond to) the FOV ofthe single ToF sensor. The ToF system can generate a depth mapcorresponding to the entire FOV by combining the depth informationobtained from each FOV portion. Thus, the resolution (e.g., imageresolution) of the depth map of the entire FOV corresponds to theresolution of the single ToF sensor multiplied by the number of scannedFOV portions. Further, because each FOV portion is smaller than thetotal FOV, the light projected by the ToF device onto each FOV portionis highly concentrated (e.g., as compared to the concentration of lightprojected by ToF systems that illuminate an entire FOV at once). Thisincrease in illumination results in a corresponding decrease in exposuretime (e.g., the amount of time required to obtain accurate distancemeasurements), thereby enabling the ToF system to maintain a high framerate while generating or updating a depth map. For example, if the ToFsystem divides an FOV into x sub-FOVs, the ToF system can generate depthmaps that have x times higher resolution with the same latency as ToFsystems that do not sequentially scan FOV portions.

In some examples, the ToF system can include a projector systemconfigured to sequentially illuminate individual FOV portions. Theprojector system can include one or more light-emitting devices. In oneexample, a light-emitting device can include a plurality of verticalcavity surface emitting lasers (VCSELs) arranged in a VCSEL array. Insome cases, each VCSEL in the VCSEL array can emit light with anarrowband near infrared (NIR) light. The projector system can alsoinclude one or more light-directing mechanisms (e.g., one or moreprojection lenses, prisms, a combination of projection lenses and one ormore prisms, and/or scanning mirrors) that appropriately direct thelight emitted by the light-emitting devices.

In one example, the projector system can include multiple light-emittingdevices (e.g., multiple VSCEL arrays), with each light-emitting devicebeing configured to illuminate a particular FOV portion of the entireFOV. For instance, the number of light-emitting devices can equal thenumber of FOV portions into which the FOV is divided. Eachlight-emitting device can be selectively and/or individually activated(e.g., such that only a single light-emitting device is emitting lighttoward a respective FOV portion at any point in time). In such anexample, the one or more light-directing mechanisms can include a lensarray of multiple projection lenses and/or a prism (or multiple prismsin some cases). For instance, each projection lens of the lens array cancorrespond to a respective light-emitting device. The multipleprojection lenses of the lens array can be positioned (e.g., held at aparticular angle) to ensure that light emitted by each light-emittingdevice illuminates an appropriate FOV portion. Using multiplelight-emitting devices in combination with a lens array and/or a prism(or multiple prisms) can ensure that no mechanical movement is necessaryto capture distance measurements for the entire FOV.

In another example, the projector system can include a singlelight-emitting device (e.g., a single VSCEL array) configured toselectively illuminate individual FOV portions. In such an example, theone or more light-directing mechanisms can include a scanning mirror(e.g., a micro electrical mechanism system (MEMS)) mirror). Theorientation of the scanning mirror can be adjusted (e.g., moved to aparticular angle) periodically to sequentially reflect light emitted bythe light-emitting device in order to illuminate each FOV portion. Whileusing a scanning mirror may involve mechanical movement of the ToFsystem, the mechanical fatigue of the scanning mirror may be negligible.Thus, the mechanical movement of the scanning mirror may not degrade theperformance of the projector system over the lifetime of the scanningmirror. Using the scanning mirror can enable only a singlelight-emitting device to be used by the ToF system.

The ToF system can generate a complete depth map of an entire FOV bycombining distance measurements corresponding to individual FOV portionsof the entire FOV. For instance, the ToF system can “stitch” togetherframes corresponding to individual FOV portions. The frames can becombined as the frames are captured, or after all of the frames havebeen captured. In some cases, frames corresponding to adjacent FOVportions can include a predetermined amount of overlap (e.g., apredetermined number of overlapping pixels). The overlap may enable theToF system to efficiently and/or seamlessly compile individual framesinto a cohesive depth map of an entire FOV.

In some cases, the ToF system can synchronize the projector system andthe ToF sensor in accordance with a synchronization scheme. Byperforming the synchronization, the ToF system can ensure that the ToFsensor determines distance measurements associated with a particular FOVportion when that particular FOV portion is illuminated. In one example,the ToF system can include a controller that sends a synchronizationsignal to the projector system. If the projector system includesmultiple light-emitting devices (e.g., multiple VCSEL arrays), thesynchronization signal can indicate which light-emitting device iscurrently to be turned on. If the projector system includes a scanningmirror, the synchronization signal can indicate a particular angle atwhich the scanning mirror is to be oriented. In addition, the controllercan send a synchronization signal to the ToF sensor. Thissynchronization signal can indicate, to the sensor, which FOV portion iscurrently being illuminated by the projector system. Based on thesynchronization signal, the ToF sensor can accurately label distancemeasurements associated with individual FOV portions as the FOV portionsare sequentially illuminated.

Further details regarding high resolution ToF systems are providedherein with respect to various figures. FIG. 1 is a diagram illustratingan example image processing system 100 for time-of-flight (ToF) signalprocessing. In this illustrative example, the image processing system100 can include a ToF system 102, an image sensor 104, a storage 106,and an application processor 110. In some examples, the image processingsystem 100 can optionally include other compute components 108 such as,for example, a central processing unit (CPU), a graphics processing unit(GPU), a digital signal processor (DSP), and/or an image signalprocessor (ISP), which the image processing system 100 can use toperform one or more of the operations/functionalities described hereinwith respect to the application processor 110. In some cases, theapplication processor 110 and/or the other compute components 108 canimplement a ToF engine 130, an image processing engine 134, and/or arendering engine 136.

It should be noted that, in some examples, the application processor 110and/or the other compute components 108 can also implement one or morecomputing engines that are not shown in FIG. 1. The ToF engine 130, theimage processing engine 134, and the rendering engine 136 are providedherein for illustration and explanation purposes and other possiblecomputing engines are not shown for the sake of simplicity. Also, forillustration and explanation purposes, the ToF engine 130, the imageprocessing engine 134, the rendering engine 136, and their variousoperations disclosed operations will be described herein as beingimplemented by the application processor 110. However, one of skill inthe art will recognize that, in other examples, the ToF engine 130, theimage processing engine 134, the rendering engine 136, and/or theirvarious operations disclosed operations can be implemented by the othercompute components 108.

The image processing system 100 can be part of, or implemented by, acomputing device or multiple computing devices. In some examples, theimage processing system 100 can be part of an electronic device (ordevices) such as a camera system (e.g., a digital camera, an IP camera,a video camera, a security camera, etc.), a telephone system (e.g., asmartphone, a cellular telephone, a conferencing system, etc.), a laptopor notebook computer, a tablet computer, a set-top box, a television, adisplay device, a digital media player, a gaming console, a videostreaming device, a head-mounted display (HMD), an extended reality (XR)device, a drone, a computer in a car, an IoT (Internet-of-Things)device, a smart wearable device, or any other suitable electronicdevice(s). In some implementations, the ToF system 102, the image sensor104, the storage 106, the other compute components 108, the applicationprocessor 110, the ToF engine 130, the image processing engine 134, andthe rendering engine 136 can be part of the same computing device.

For example, in some cases, the ToF system 102, the image sensor 104,the storage 106, the other compute components 108, the applicationprocessor 110, the ToF engine 130, the image processing engine 134, andthe rendering engine 136 can be integrated into a camera, smartphone,laptop, tablet computer, smart wearable device, HMD, XR device, IoTdevice, gaming system, and/or any other computing device. However, insome implementations, one or more of the ToF system 102, the imagesensor 104, the storage 106, the other compute components 108, theapplication processor 110, the ToF engine 130, the image processingengine 134, and/or the rendering engine 136 can be part of, orimplemented by, two or more separate computing devices.

The ToF system 102 can use light, such as near infrared light (NIR), todetermine depth and/or distance information about a target (e.g., asurrounding/nearby scene, one or more surrounding/nearby objects, etc.).In some examples, the ToF system 102 can measure both the distance andintensity of each pixel in a target such as a scene. The ToF system 102can include a light emitter to emit a light signal towards a target(e.g., a scene, an object, etc.), which can hit the target andreturn/reflect to the ToF system 102. The ToF system 102 can include asensor to detect and/or measure the returned/reflected light, which canthen be used to determine depth and/or distance information for thetarget. The distance of the target relative to the ToF system 102 can beused to perform depth mapping. The distance of the target can becalculated through direct ToF or indirect ToF.

In direct ToF, the distance can be calculated based on the travel timeof the emitted light pulse signal and the returned/reflected light pulsesignal (e.g., the time from when the light signal was emitted and thereturned/reflected light signal was received). For example, theround-trip distance of the emitted light signal and thereturned/reflected light signal can be calculated by multiplying thetravel time of the emitted light pulse signal and the returned/reflectedlight pulse signal by the speed of light, commonly denoted c. Theround-trip distance calculated can then be divided by 2 to determine thedistance from the ToF system 102 to the target.

In indirect ToF, the distance can be calculated by sending modulatedlight toward a target and measuring the phase of the returned/reflectedlight. Knowing the frequency (f) of the emitted light, the phase shiftof the returned/reflected light, and the speed of light allows thedistance to the target to be calculated. For example, runtimedifferences between the path of the emitted light and the path of thereturned/reflected light result in a phase shift of thereturned/reflected light. The phase difference between the emitted lightand the returned/reflected light and the modulation frequency (f) of thelight can be used to calculate the distance between the ToF system 102and the target. For example, the formula for the distance between theToF system 102 and the target can be (c/2f)×Phase Shift/2π. As thisshows, a higher frequency of light can provide a higher measurementaccuracy but will result in a shorter maximum distance that can bemeasured.

Accordingly, in some examples, dual frequencies can be used to improvethe measuring accuracy and/or distance, as further explained herein. Forexample, a 60 MHz light signal can be used to measure a target 2.5meters away, and a 100 MHz light signal can be used to measure a target1.5 meters away. In a dual frequency scenario, both the 60 MHz and the100 MHz light signals can be used to calculate a distance to a target.

The image sensor 104 can include any image and/or video sensor orcapturing device, such as a digital camera sensor, a video camerasensor, a smartphone camera sensor, an image/video capture device on anelectronic apparatus such as a television or computer, a camera, etc. Insome cases, the image sensor 104 can be part of a camera or computingdevice such as a digital camera, a video camera, an IP camera, asmartphone, a smart television, a game system, etc. In some examples,the image sensor 104 can include multiple image sensors, such as rearand front sensor devices, and can be part of a dual-camera or othermulti-camera assembly (e.g., including two camera, three cameras, fourcameras, or other number of cameras). The image sensor 104 can captureimage and/or video frames (e.g., raw image and/or video data), which canthen be processed by the application processor 110, the ToF engine 130,the image processing engine 134, and/or the rendering engine 136, asfurther described herein.

The storage 106 can be any storage device(s) for storing data. Moreover,the storage 106 can store data from any of the components of the imageprocessing system 100. For example, the storage 106 can store data fromthe ToF system 102 (e.g., ToF sensor data or measurements), the imagesensor 104 (e.g., frames, videos, etc.), data from and/or used by theother compute components 108 and/or the application processor 110 (e.g.,processing parameters, image data, ToF measurements, depth maps, tuningparameters, processing outputs, software, files, settings, etc.), datafrom and/or used by the ToF engine 130 (e.g., one or more neuralnetworks, image data, tuning parameters, auxiliary metadata, ToF sensordata, ToF measurements, depth maps, training datasets, etc.), imageprocessing engine 134 (e.g., image processing data and/or parameters,etc.), data from and/or used by the rendering engine 136 (e.g., outputframes), an operating system of the image processing system 100,software of the image processing system 100, and/or any other type ofdata.

The application processor 110 can include, for example and withoutlimitation, a CPU 112, a GPU 114, a DSP 116, and/or an ISP 118, whichthe application processor 110 can use to perform various computeoperations such as image/video processing, ToF signal processing,graphics rendering, machine learning, data processing, calculations,and/or any other operations. In the example shown in FIG. 1, theapplication processor 110 implements a ToF engine 130, an imageprocessing engine 134, and a rendering engine 136. In other examples,the application processor 110 can also implement one or more otherprocessing engines. Moreover, in some cases, the ToF engine 130 canimplement one or more machine learning algorithms (e.g., one or moreneural networks) configured to perform ToF signal processing and/orgenerate depth maps.

In some cases, the application processor 110 can also include a memory122 (e.g., random access memory (RAM), dynamic RAM, etc.) and a cache120. The memory 122 can include one or more memory devices, and caninclude any type of memory such as, for example, volatile memory (e.g.,RAM, DRAM, SDRAM, DDR, static RAM, etc.), flash memory, flashed-basedmemory (e.g., solid-state drive), etc. In some examples, the memory 122can include one or more DDR (e.g., DDR, DDR2, DDR3, DDR4, etc.) memorymodules. In other examples, the memory 122 can include other types ofmemory module(s). The memory 122 can be used to store data such as, forexample, image data, ToF data, processing parameters (e.g., ToFparameters, tuning parameters, etc.), metadata, and/or any type of data.In some examples, the memory 122 can be used to store data from and/orused by the ToF system 102, the image sensor 104, storage 106, the othercompute components 108, the application processor 110, the ToF engine130, the image processing engine 134, and/or the rendering engine 136.

The cache 120 can include one or more hardware and/or softwarecomponents that store data so that future requests for that data can beserved faster than if stored on the memory 122 or storage 106. Forexample, the cache 120 can include any type of cache or buffer such as,for example, system cache or L2 cache. The cache 120 can be fasterand/or more cost effective than the memory 122 and storage 106.Moreover, the cache 120 can have a lower power and/or operational demandor footprint than the memory 122 and storage 106. Thus, in some cases,the cache 120 can be used to store/buffer and quickly serve certaintypes of data expected to be processed and/or requested in the future byone or more components (e.g., application processor 110) of the imageprocessing system 100, such as image data or ToF data.

In some examples, the operations for the ToF engine 130, the imageprocessing engine 134, and the rendering engine 136 (and any otherprocessing engines) can be implemented by any of the compute componentsin the application processor 110. In one illustrative example, theoperations of the rendering engine 136 can be implemented by the GPU114, and the operations of the ToF engine 130, the image processingengine 134, and/or one or more other processing engines can beimplemented by the CPU 112, the DSP 116, and/or the ISP 118. In someexamples, the operations of the ToF engine 130, and the image processingengine 134 can be implemented by the ISP 118. In other examples, theoperations of the ToF engine 130, and/or the image processing engine 134can be implemented by the ISP 118, the CPU 112, the DSP 116, and/or acombination of the ISP 118, the CPU 112, and the DSP 116.

In some cases, the application processor 110 can include otherelectronic circuits or hardware, computer software, firmware, or anycombination thereof, to perform any of the various operations describedherein. In some examples, the ISP 118 can receive data (e.g., imagedata, ToF data, etc.) captured or generated by the ToF system 102 and/orthe image sensor 104 and process the data to generate output depth mapsand/or frames. A frame can include a video frame of a video sequence ora still image. A frame can include a pixel array representing a scene.For example, a frame can be a red-green-blue (RGB) frame having red,green, and blue color components per pixel; a luma, chroma-red,chroma-blue (YCbCr) frame having a luma component and two chroma (color)components (chroma-red and chroma-blue) per pixel; or any other suitabletype of color or monochrome picture.

In some examples, the ISP 118 can implement one or more processingengines (e.g., ToF engine 130, image processing engine 134, etc.) andcan perform ToF signal processing and/or image processing operations,such as depth calculation, depth mapping, filtering, demosaicing,scaling, color correction, color conversion, noise reduction filtering,spatial filtering, artifact correction, etc. The ISP 118 can processdata from the ToF system 102, the image sensor 104, storage 106, memory122, cache 120, other components in the application processor 110,and/or data received from a remote source, such as a remote camera, aserver or a content provider.

While the image processing system 100 is shown to include certaincomponents, one of ordinary skill will appreciate that the imageprocessing system 100 can include more or fewer components than thoseshown in FIG. 1. For example, the image processing system 100 can alsoinclude, in some instances, one or more other memory devices (e.g., RAM,ROM, cache, and/or the like), one or more networking interfaces (e.g.,wired and/or wireless communications interfaces and the like), one ormore display devices, and/or other hardware or processing devices thatare not shown in FIG. 1. An illustrative example of a computing deviceand hardware components that can be implemented with the imageprocessing system 100 is described below with respect to FIG. 10.

FIG. 2A is a simplified block diagram illustrating an example of adirect ToF sensing procedure 200. In this example, the ToF system 102first emits light pulse 202 towards a target 210. The target 210 caninclude, for example, a scene, one or more objects, one or more animals,one or more people, etc. The light pulse 202 can travel to the target210 until it hits the target 210. When the light pulse 202 hits thetarget 210, at least some portion of the light pulse 202 can bereflected back to the ToF system 102.

Thus, the ToF system 102 can receive reflected light pulse 204 includingat least some portion of the light pulse 202 reflected back from thetarget 210. The ToF system 102 can sense the reflected light pulse 204and calculate the distance 206 to the target 210 based on the reflectedlight pulse 204. To calculate the distance 206, the ToF system 102 cancalculate the total time traveled by the emitted light pulse 202 and thereflected light 204 (e.g., the time from when the light pulse 202 wasemitted to when the reflected light pulse 204 was received). The ToFsystem 102 can multiply the total time traveled by the emitted lightpulse 202 and the reflected light pulse 204 by the speed of light (c) todetermine the total distance traveled by the light pulse 202 and thereflected light pulse 204 (e.g., the round trip time). The ToF system102 can then divide the total time traveled by 2 to obtain the distance206 from the ToF system 102 to the target 210.

FIG. 2B is a simplified block diagram illustrating an example of anindirect ToF sensing procedure 220. In this example, the phase shift ofreflected light can be calculated to determine depth and distance forthe target 210. Here, the ToF system 102 first emits modulated light 222towards the target 210. The modulated light 222 can have a certain knownor predetermined frequency. The modulated light 222 can travel to thetarget 210 until it hits the target 210. When the modulated light 222hits the target 210, at least some portion of the modulated light 222can be reflected back to the ToF system 102.

The ToF system 102 can receive the reflected light 224 and determine thephase shift 226 of the reflected light 224 and the distance 206 to thetarget 210 using the formula distance (206)=(c/2f)×Phase Shift/2π, wheref is the frequency of the modulated light 222 and c is the speed oflight.

In some cases, when calculating depth and distance (e.g., 206), one ormore factors that affect how the light is reflected can be taken intoaccount or used to tune the calculations. For example, objects andsurfaces can have specific characteristics which can cause light toreflect differently. To illustrate, different surfaces can havedifferent indexes of refraction, which can affect how light travels orinterfaces with the surfaces and/or the material(s) in the surfaces.Moreover, non-uniformities, such as material irregularities orscattering centers, can cause light to be reflected, refracted,transmitted, or absorbed, and can sometimes cause loss of energy. Thus,when light hits a surface, it can be absorbed, reflected, transmitted,etc. The proportion of light reflected by the surface is called itsreflectance. However, the reflectance does not only depend on thesurface (e.g., index of refraction, material properties, uniformities ornon-uniformities, etc.), but can also depend on the type of light beingreflected and the surrounding environment (e.g., temperature, ambientlight, water vapor, etc.). Therefore, as further explained below, insome cases, information about the surrounding environment, the type oflight, and/or characteristics of the target 210 can be factored in whencalculating the distance 206 and/or depth information for the target210.

FIG. 3 is a diagram illustrating an example technique for measuring thephase angle of a light signal. In this example, the ToF system 102 emitsa modulated light signal 222 and receives a reflected light 224. The ToFsystem 102 measures the amplitude (A) of the reflected light 224 at fourdifferent points 302, 304, 306, and 308, to determine amplitude A₁ atmeasurement point 302, amplitude A₂ at measurement point 304, amplitudeA₃ at measurement point 306, and amplitude A₄ at measurement point 308.The measurement points 302, 304, 306, and 308 can be equally spacedpoints (e.g., 0°, 90°, 180°, 270°).

The phase angle of the reflected light 224 can be represented by thefollowing equation:

$\begin{matrix}{\varphi = {{ArcTan}\left( \frac{A_{2} - A_{4}}{A_{1} - A_{3}} \right)}} & {{Eq}.1}\end{matrix}$

Equation 1 illustrates the relationship between the amplitudemeasurements at points 302, 304, 306, and 308 and the phase angle of thereflected light 224. The ratio of the difference between A₁ and A₃ andthe difference between A₂ and A₄ is equal to the tangent of the phaseangle.

FIG. 4 is a diagram illustrating a continuous wave method 400 for ToFsensing. In this example, the light signal 402 emitted and the reflectedlight signal 404 can be cross-correlated to determine a phase delay ofthe reflected light signal 404. The light signal 402 can first beemitted and sampled quantities 406-412 (Q₁, Q₂, Q₃, Q₄) of the reflectedlight signal 404 can be measured using four out-of-phase time windows(e.g., sensor integration time windows) C₁ 414, C₂ 416, C₃ 418 and C₄420, with each window (414-420) being phase-stepped such that C₁ is 0degrees, C₂ is 180 degrees, C₃ is 90 degrees, and C₄ is 270 degrees. Asillustrated in FIG. 4, the measured quantities 406-412 correspond to theoverlap regions between the measurement time windows and the reflectedlight signal 404.

The phase delay between the light signal 402 emitted and the reflectedlight signal 404, φ, and the distance, d, can then be calculated by thefollowing equations:

$\begin{matrix}{\varphi = {{ArcTan}\left( \frac{Q_{3} - Q_{4}}{Q_{1} - Q_{2}} \right)}} & {{Eq}.2}\end{matrix}$ $\begin{matrix}{d = {\frac{c}{4\pi f}\varphi}} & {{Eq}.3}\end{matrix}$

In some examples, the quantities 406-412 (Q₁, Q₂, Q₃, Q₄) can be used tocompute pixel intensity and offset. Moreover, in some cases, the terms(Q₃−Q₄) and (Q₁−Q₂) can reduce the effect of constant offset from thequantities (406-412). The quotient in the phase equation, equation 2,can reduce the effects of system or environmental variations from thedistance measurements such as, for example, system amplification, systemattenuation, the reflected intensity, etc.

FIG. 5 is a diagram illustrating an example image processing system 500for ToF signal processing. In some cases, the image processing system500 can correspond to all or a portion of the image processing system100 of FIG. 1. As shown in FIG. 5, the image processing system 500 caninclude a projection system 502, a receiving system 508, and a processor518. These and/or other components of the image processing system 500can be configured to generate a depth map 516. The depth map 516 caninclude depth information associated with targets (e.g., objects) withina FOV of the image processing system 500.

In some cases, the projection system 502 can include one or morelight-emitting devices 504. The one or more light-emitting devices 504can include any device configured to emit light for the purpose ofobtaining depth information (e.g., distance measurements) for one ormore targets. In an illustrative example, the one or more light-emittingdevices 504 can include one or more vertical cavity surface emittinglasers (VCSELs) or other laser devices. For instance, the one or morelight-emitting devices 504 can include a plurality of VCSELs arranged ina VCSEL array. As will be explained in greater detail below, theprojection system 502 can include a single VCSEL array, or multipleVCSEL arrays. In some cases, the one or more light-emitting devices 504can be configured to emit light with a certain wavelength (or certainrange of wavelengths). In a non-limiting example, the one or morelight-emitting devices 504 can be configured to emit narrowband nearinfrared (NIR) light (e.g., light with a wavelength between 800 and 2500nanometers). The one or more light-emitting devices 504 can emit lightwith any wavelength suitable for obtaining ToF data.

As shown, the projection system 502 can include one or morelight-directing devices 506. The one or more light-directing devices 506can include any device configured to change, adjust, or otherwise directthe angle of light emitted by the one or more light-emitting devices504. In one example, the one or more light-directing devices 506 caninclude one or more projection lenses that disperse light by means ofrefraction. In another example, the light-directing devices 506 caninclude one or more mirrors that reflect light at a desired angle basedon the law of mirror reflection. As will be explained in more detailbelow, the one or more light-directing devices 506 can be configured toselectively direct light emitted by the one or more light-emittingdevices 504 to particular portions of the FOV of the image processingsystem 500.

In some cases, the projection system 502 can generate and projectillumination signals 512. The illumination signals 512 can correspond tolight emitted by the one or more light-emitting devices 504 and directedby the one or more light-directing devices 506. In one example, theillumination signals 512 can include a plurality of illumination signalsemitted at different points in time. For instance, the illuminationsignals 512 can include a series of illumination signals that aresequentially directed to different FOV portions of the image processingsystem 500. The illumination signals 512 can be absorbed, transmitted,and/or reflected by one or more targets. All or a portion of theillumination signals that are reflected can be captured by the receivingsystem 508. The reflected and captured illumination signals cancorrespond to illumination reflections 514 in FIG. 5. In one example,the illumination reflections 514 can be received by a sensor 510 of thereceiving system 508. The sensor 510 can include any type or form ofdevice configured to receive, capture, and/or process light having thewavelength of the light emitted by the one or more light-emittingdevices 504. In one example, the sensor 510 can include a spectrumbandpass filter that filters all or a portion (e.g., most) of theambient interference while passing light having a wavelengthapproximately the same as (e.g., within certain tolerance of) theillumination signals 512. In some cases, the sensor 510 can include aToF sensor. For example, the sensor 510 can correspond to all or aportion of the ToF system 102 and/or the image sensor 104 of the imageprocessing system 100. The sensor 510 can process the illuminationreflections 514 to determine distance information associated with one ormore targets. For example, the sensor 510 can determine the distance toa target based on an amount of time between projection of anillumination signal and receipt of a corresponding illuminationreflection (in accordance with the direct ToF technique describedabove). In another example, the sensor 510 can determine the distance toa target based on a phase shift between an illumination signal and acorresponding illumination reflection (in accordance with the indirectToF technique described above). In some cases, the resolution (e.g.,image resolution) of the sensor 510 corresponds to the number of pixels(e.g., photo-sensitive elements) within the sensor 510. All or a portionof the pixels of the sensor 510 can receive an illumination reflectionin response to a projected illumination signal. In one example, thesensor 510 can determine a distance measurement corresponding to eachpixel of the sensor 510.

In some examples, the processor 518 can generate the depth map 516 basedat least in part on the distance measurements determined by the sensor510. For example, the sensor 510 and/or the processor 518 can combinedistance measurements associated with individual FOV portions togenerate a complete set of distance measurements for the entire FOV ofthe image processing system 500. The processor 518 can determine a depthmap for the entire FOV using the complete set of distance measurements.

FIG. 6A is a diagram of an example FOV 602 of the image processingsystem 500 of FIG. 5. In this example, the FOV 602 can correspond to thecumulative FOV of the projection system 502. For example, the FOV 602can represent the total divergence angle of the projection system 502.As described above, the one or more light-directing devices 506 can beconfigured to direct the light emitted by the one or more light-emittingdevices 504 at various angles (e.g., in order to reduce the divergenceangle of the projection system 502 to a particular FOV portion). In oneexample, each FOV portion can be associated with a particularconfiguration of the components of the projection system 502. Thus, theFOV 602 can represent a combination of the FOVs resulting from eachpossible projection configuration of the projection system 502.

FIG. 6B is a diagram of example FOV portions of the FOV 602. In thisexample, the FOV 602 can include 9 FOV portions. In some cases, theresolution (e.g., image resolution) of ToF data associated with each FOVportion can correspond to the full resolution of the sensor 510. Forexample, the scene of each FOV portion is imaged sequentially on theentire sensor 510. In this way, the full resolution of the sensor 510can be utilized at each configuration. Further, the full illuminationpower of the projection system 502 can be utilized at eachconfiguration. As will be explained in more detail below, the imageprocessing system 500 can combine ToF data associated with multiple FOVportions to generate a depth map that has a resolution greater than theresolution of the sensor 510. For example, the resolution of the depthmap 516 can correspond to the resolution of the sensor 510 multiplied bythe number of FOV portions into which the FOV 602 is divided. In anillustrative example, the resolution of the sensor 510 can be given asr=n×m and the FOV 602 can be divided into N×M FOV portions. In thisexample, the resolution of a cumulative depth map corresponding to theentire FOV can be given as R=Nn×Mm. Further, if one FOV portioncorresponds to a horizontal projection angle θ_(x) and a verticalprojection angle θ_(y), the horizontal and vertical projection angles ofthe FOV 602 can be given as θ_(x)=Nθ_(x) and θ_(Y)=Mθ_(y), respectively.See FIG. 6A and FIG. 6B for reference.

In some cases, the projection system 502 can sequentially illuminateeach FOV portion of the FOV 602. For example, the projection system 502can illuminate a first FOV portion by projecting a first illuminationsignal while in a first projection configuration. The receiving system508 can capture and process the first illumination reflection with theentire sensor 510 resolution, and the full resolution ToF frame of thefirst illumination is stored in the memory buffer. After the firstillumination reflection is received and/or processed, the projectionsystem 502 can illuminate a second FOV portion by projecting a secondillumination signal while in a second projection configuration. Thereceiving system 508 can capture and process the second illuminationreflection with the entire sensor 510 resolution, and the fullresolution ToF frame of the second illumination is also stored in thememory buffer. In some examples, the image processing system 500 canrepeat the process of projecting illumination signals and receivingillumination reflections until an illumination reflection correspondingto each FOV portion has been received. During this process, a single FOVportion can be illuminated at a time. For instance, the projectionsystem 502 can be configured to stop projecting light towards one FOVportion before projecting light towards another FOV portion.

In one example, the projection system 502 can illuminate FOV portions inaccordance with a sequential illumination order. The sequentialillumination order can correspond to a predetermined order in which theprojection system 502 “scans” the FOV portions. A sequentialillumination order can include scanning through rows of FOV portions,scanning through columns of FOV portions, scanning diagonally across FOVportions, among other illumination orders. In an illustrative example, asequential illumination order for illuminating the FOV portions shown inFIG. 6B can follow the sequence of ToF frames FOV_(1,1), FOV_(1,2),FOV_(1,3), FOV_(2,1), etc. In some cases, the receiving system 508 canreceive illumination reflections in a sequential receiving order thatcorresponds to the sequential illumination order. For example, thesensor 510 can determine distance measurements in response to eachillumination reflection received by the sensor 510. The sensor 510 canrecord particular distance measurements in association with appropriateFOV portions based on the sequential illumination order. For example,the image processing system 500 can send a synchronization signal to theprojection system 502 and the receiving system 508. The synchronizationsignal can indicate, to both the projection system 502 and the receivingsystem 508, the FOV currently being illuminated.

As mentioned above, the image processing system 500 can combine (e.g.,“stitch together”) distance measurements associated with individual FOVportions to generate a complete set of distance measurements for theentire FOV. To combine distance measurements associated with individualFOV portions, the image processing system 500 can store ToF frames foreach FOV portion (e.g., ToF frames FOV_(1,1), FOV_(1,2), FOV_(1,3),FOV_(2,1), through FOV_(3,3) for the FOV portions of FIG. 6B). In oneexample, the image processing system 500 can store the ToF frames in abuilt-in memory buffer of the sensor 510. In another example, the imageprocessing system 500 can store the ToF frames in a system memoryexternal to the sensor 510 (e.g., the storage 106 of the imageprocessing system 100). In some cases, the image processing system 500can combine distance measurements associated with FOV portions inreal-time. For example, the sensor 510 and/or the processor 518 canincorporate distance measurements into the complete set of distancemeasurements as new illumination reflections are received and processedin accordance with the sequential illumination order. In other examples,the sensor 510 and/or the processor 518 can generate the complete set ofdistance measurements after each FOV portion has been scanned. Thecomplete set of distance measurements can be stored in a single location(e.g., the same file or data structure), and used for generating thedepth map 516. The image stitching operation can be implemented by thesensor electronics of the image processing system 500 or outside of thesensor electronics (e.g., by the device CPU or other processor).

In some examples, one FOV portion can partially overlap another FOVportion. For example, FIG. 6B illustrates an example overlap 612 betweenFOV_(1,1) and FOV_(1,2). In one example, the overlap 612 can correspondto an overlap of a predetermined number of pixels. For instance, thehorizontal resolution n of FOV_(1,1) can be given by n=n_(x)+x, wheren_(x) corresponds to pixels unique to FOV_(1,1) and x corresponds topixels included in both FOV_(1,1) and FOV_(1,2). Similarly, FOV_(1,1)and FOV_(2,1) can share an overlap 614. In one example, the verticalresolution m of FOV_(1,1) can be given by m=m_(y)+y, where m_(y)corresponds to pixels unique to FOV_(1,1) and y corresponds to pixelsincluded in both FOV_(1,1) and FOV_(2,1). As shown in FIG. 6B, all or aportion of the remaining FOV portions can include similar overlaps. Insome cases, overlaps between FOV portions can facilitate implementing animage stich scheme that seamlessly “stitches” the FOV portions togetherusing an automated digital alignment technique. For example, the imageprocessing system 500 can localize pixels within FOV portions relativeto the entire FOV based at least in part on detecting overlapping (e.g.,matching) distance measurements. The image processing system 500 canimplement overlaps of any suitable size, including no overlap.

FIG. 7 is a diagram of a device 700 corresponding to an exampleimplementation of the projection system 502 of FIG. 5. In some examples,the device 700 can include multiple light-emitting devices. In anillustrative example, the device 700 can have a plurality of VCSELarrays. As shown, the device 700 can include one or more light-emittingdevices, including light-emitting devices 708(A), 708(B), and 708(C).Although FIG. 7 illustrates three light-emitting devices 708(A), 708(B),and 708(C), the device 700 can include any suitable number oflight-emitting devices arranged in any suitable configuration. In anillustrative example, the device 700 can include nine light-emittingdevices arranged in a 3×3 grid. In some cases, the number oflight-emitting devices of device 700 can correspond to the number of FOVportions into which the FOV 602 is divided. In these cases, eachlight-emitting device can correspond to a different FOV portion. Forexample, the projection system 502 can sequentially activate (e.g., turnon) each light-emitting device in turn in order to scan each FOVportion.

In some cases, the device 700 can include one or more projection lensesand/or additional light-directing components configured to project thelight emitted by the light-emitting devices. For example, the device 700can include a plurality of projection lenses (e.g., a projection lensarray including projection lenses 710(A), 710(B), and/or 710(C)). Insome cases, each projection lens of the projection lens array cancorrespond to one of the light-emitting devices (e.g., light from thelight-emitting device 708(A) is emitted through a correspondingprojection lens 710(A) of the plurality of projection lenses, light fromthe light-emitting device 708(B) is emitted through a correspondingprojection lens 710(B), and so on). In an illustrative example, theprojection lens array can include a plurality of discrete (e.g.,separated) projection lenses. In some examples, the device 700 does notinclude a projection lens array (i.e., the projection lens array can beoptional). In some cases, the focal length of the projection lenses710(A), 710(B), and 710(C) of the projection lens array determines thedegrees of each of the FOV portions, e.g., FOV_(1,1), FOV_(1,2), . . .FOV_(n,m), . . . . In some cases, the focal length of the projectionlenses 710(A), 710(B), and 710(C) together with the light directingelement (e.g., the prism 714) determines the amount of overlappingbetween the FOV portions.

In some cases, a prism 714 shown in FIG. 7 can be configured to directlight at different angles (or a different range of angles). In someexamples, each projection lens of the projection lens array, togetherwith the prism 714, can be configured to direct light at a particularangle. For example, when one light-emitting device 708(A), 708(B), or708(C) is activated, the prism 714 (or a projection lens-prism assemblycorresponding to a particular light-emitting device) can bend light froma light-emitting device (e.g., light-emitting device 708(A), 708(B), or708(C)) towards a desired FOV portion. In one example, the prism 714 canbe a segmented prism array that includes multiple prism segmentsconfigured to refract light in different directions. For example, eachprism on the segmented prism array can bend light from a correspondinglight-emitting device to a desired angle associated with a particularFOV portion. While a prism 714 is shown in FIG. 7, another type ofoptical element can be used other than a prism, such as a diffractiveoptical element (DOE). The direction in which a prism segment (or otheroptical element) refracts light can depend on the refractive index ofthe prism segment, the apex angle of the prism segment (e.g., the anglebetween two faces of the prism segment), among other characteristics. Insome cases, the characteristics of a prism segment corresponding to alight-emitting device (e.g., the prism segment positioned above thelight-emitting device) can be selected such that the prism segment bendslight emitted by the light-emitting device towards a desired FOVportion.

In some examples, the device 700 can include a plurality of diffusers(e.g., a diffuser array 712). Each diffuser of the diffuser array 712can correspond to (e.g., be positioned above) a respective projectionlens of the projection lens array (e.g., a first diffuser aboveprojection lens 710(A), a second diffuser above projection lens 710(B),and a third diffuser above projection lens 710(C)). In some cases, thediffuser array 712 (e.g., together with the projection lens array) candiffuse the light emitting from the pixelated VCSEL (or other type oflight source) and increase the uniformity of illumination projectionsgenerated by the device 700. In some cases, the diffusers 712 can beplaced on the other side of the projection lenses 710(A), 710(B), and710(C) of the projection lens array, on top of the VCSEL array 708, orin any other suitable configuration. The device 700 can include anyadditional or alternative light-directing devices not illustrated inFIG. 7.

FIG. 8 is a diagram of a device 800 corresponding to another exampleimplementation of the projection system 502 of FIG. 5. In some examples,the device 800 can include a single light-emitting device 808. In anillustrative example, the light-emitting device 808 can be a singleVCSEL array. The single light-emitting device 808 can be used tosequentially illuminate each FOV portion into which a full FOV isdivided. In some cases, the light emitted by the light-emitting device808 can be directed by a scanning mirror 814. In an illustrativeexample, the scanning mirror 814 can be a micro electro mechanism system(MEMS) mirror. A MEMS mirror scanner can provide various benefits,including high reliability, a long lifetime, a compact size, low cost,among other benefits. The scanning mirror 814 can include and/orcorrespond to any additional or alternative type of scanning mirror.

In some cases, the device 800 can adjust the orientation of the scanningmirror 814 in order to adjust the direction of the emitted light. Forexample, the scanning mirror 814 can have a plurality of orientations.Each orientation can correspond to an illumination of a different FOVportion. In some cases, a particular orientation of the scanning mirror814 can correspond to a particular angle between the scanning mirror 814and a reference plane. In one example, the reference plane can be oneplane (e.g., one face) of the light-emitting device 808. FIG. 8illustrates three example orientations of the scanning mirror 814. Anorientation O₁ of the scanning mirror 814 is illustrated in FIG. 8 witha dashed line. The direction of light projected by the scanning mirror814 at orientation O₁ is illustrated with corresponding dashed arrows.Orientations O₂ and O₃ (and their corresponding directions of projectedlight) are illustrated with solid and dotted lines, respectively.

The device 800 can include one or more additional light-directingdevices, such as a projection lens 810 and/or a diffuser 812. In somecases, the diffuser 812 (e.g., together with the lens 810 in some cases)can increase the uniformity of illumination projections generated by thedevice 800. In some cases, the diffuser 812 can be placed on the otherside of the projection lens 810 or on top of the light emitting device808 (e.g., a VCSEL array). The focal length of the projection lens 810determines the degrees of each of the FOV portions, e.g., FOV_(1,1),FOV_(1,2), . . . FOV_(n,m), . . . . In some cases, the focal length ofthe projection lens 810 together with the angles of the light directingmirror 814 determines the amount of overlapping between the FOVportions. The device 800 can include any additional or alternativelight-directing devices not illustrated in FIG. 8. In some examples, thedevice 800 does not include a projection lens 810 (i.e., the projectionlens 810 can be optional).

FIG. 9A is a diagram of a device 900 corresponding to an exampleimplementation of the receiving system 508 of FIG. 5. In some examples,the device 900 can include a sensor 910 (e.g., corresponding to thesensor 510 of FIG. 5). The device 900 can also include one or morecomponents configured to focus illumination reflections (e.g.,illumination reflections 514 based on illumination signals 512 emittedby the projection system 502 of FIG. 5, the device 700 of FIG. 7, or thedevice 800 of FIG. 8) onto the sensor 910. For instance, the device 900can include an image lens array 902 that includes one or more imagelenses (e.g., image lenses 902(A), 902(B), and/or 902(C)). In anillustrative example, the image lens array 902 can include a 3×3 matrixof image lenses. In some cases, the refractive characteristics (e.g.,refractive indices, angles, slopes, curvatures, tilts, etc.) of theimage lens array 902 can be selected to focus an illumination reflectioncorresponding to one FOV portion onto the sensor 910. Further, thesensor 910 can be positioned at a distance from the image lens array 902(or other image lens array not shown in FIG. 9) that facilitates captureof an illumination reflection. For example, the sensor 910 can bepositioned within the back focal plane of the image lens array 902. Inone illustrative example, the image lens array 902 can have a focallength f. The sensor 910 can be positioned at a distance f from theimage lens array 902 to facilitate and/or ensure that the sceneilluminated by the projector of FIG. 7 or FIG. 8 is focused or imaged onthe sensor 910. Further, in some cases, the sensor 910 can be centeredrelative to the middle and/or center lens segment of the image lensarray 902. In these cases, the center-to-center distance between theimage lens segments can be designed to facilitate illuminating FOVportions of particular sizes (e.g., particular projection angles) and/orto facilitate focus of illumination reflections. For example, thelocations of the image lenses 902(A), 902(B), and/or 902(C) can bedesigned to facilitate a desired level of light-dispersal and/orlight-focusing. Each image lens of the image lens array 902 (the imagelenses 902(A), 902(B), and 902(C)) is designated to image the portion ofthe FOV, i.e., FOV_(1,1), FOV_(1,2), FOV_(n,m), . . . to the sensor 910.In some cases, the number of the image lenses is the same as the numberof FOV portions.

In some examples, the image lens array 902 can be configured foroff-axis imaging. For example, in an off-axis imaging configuration, theoptical axis of an image lens is not aligned with the center of thesensor. In some cases, such off-axis imaging can result in aberrations(e.g., undesirable dispersal of light). To correct and/or account foraberrations associated with off-axis imaging, one or more image lenseswithin the image lens array 902 can be tilted relative to each other, asillustrated in FIG. 9A. For example, the sensor 910 may be aligned with(e.g., positioned directly below) the center image lens 902(B) of theimage lens array 902 and the other image lenses of the image lens array902 (e.g., the image lenses 902(A) and 902(C)) may be misalignedrelative to the sensor 910).

In some cases, the device 900 can include one or more filters thatfacilitate capture of illumination reflections. For instance, the device900 can include a bandpass filter configured to pass light with afrequency corresponding to the frequency of light emitted by one or morelight-emitting devices (e.g., the light-emitting devices of the device700 of FIG. 7 and/or the device 800 of FIG. 8). In an illustrativeexample, the device 900 can include a narrow bandpass filter with acenter frequency corresponding to NIR light. The bandpass filter canpass light corresponding to illumination reflections while at leastpartially blocking light interference from external light sources, e.g.,sunlight, thereby increasing the signal-to-noise or signal-to-backgroundnoise ratio, and consequently, increasing the accuracy and/or quality ofgenerated depth maps. FIG. 9A illustrates an example filter 922 that canbe implemented within the device 900. As shown, the filter 922 can bepositioned above the sensor 910.

Together with a light projection system (e.g., the projection system 502of FIG. 5, which can include the device 700 of FIG. 7 and/or the device800 of FIG. 8), the device 900 can perform a sequential ToF imagecapturing process. FIG. 9B and FIG. 9C illustrate example sequential ToFimage capturing processes that can be performed by the device 900 andthe devices 700 and/or 800. The following description provides anexample of a sequential ToF image capturing process performed by thedevice 900 and the device 700. At a time T₁, the device 700 can activatethe light-emitting device 708(A), causing light to be directed towards afirst FOV portion by the prism 714 and/or the corresponding projectionlens 710(A) of the projection lens array. In response (e.g., at a timesynchronized with time T₁), the sensor 910 can receive lightcorresponding to an illumination reflection of the first FOV portion(e.g., a FOV portion corresponding to the ToF frame FOV_(1,1) of FIG.6B). This light is illustrated as light L₁ in FIG. 9C. As shown in FIG.9B and FIG. 9C, the received light L₁ can be directed to the sensor 910by the image lens 902(A) and/or by an additional light-directingcomponents of the device 900. At a time T₂, the device 700 can activatethe light-emitting device 708(B), causing light to be directed towards asecond FOV portion by the prism 714 and/or the corresponding projectionlens 710(B). In response (e.g., at a time synchronized with time T₂),the sensor 910 can receive light corresponding to an illuminationreflection of the second FOV portion (e.g., a FOV portion correspondingto the ToF frame FOV_(1,2) of FIG. 6B). This light is illustrated aslight L₂ in FIG. 9C. The received light L₂ can be directed to the sensor910 by the image lens 902(B) and/or by an additional light-directingcomponents of the device 900. At a time T₃, the device 700 can activatethe light-emitting device 708(C), causing light to be directed towards athird FOV portion by the prism 714 and/or the corresponding projectionlens 710(C). In response (e.g., at a time synchronized with time T₃),the sensor 910 can receive light corresponding to an illuminationreflection of the third FOV portion (e.g., a FOV portion correspondingto the ToF frame FOV_(1,3) of FIG. 6B). This light is illustrated aslight L₃ in FIG. 9C. The received light L₃ can be directed to the sensor910 by the image lens 902(C) and/or by an additional light-directingcomponents of the device 900.

In some cases, the amount of time between time T₁ and time T₂ (andbetween time T₂ and time T₃) can be sufficient for the device 700 toproject an illumination signal to one FOV, the device 900 to receive anillumination reflection corresponding to the illumination signal, andthe device 900 (and/or an additional processor or computing device) toprocess the illumination reflection for incorporation into a depth map.In some examples, the sequential ToF illumination process can bedirected at least in part by a driver 702 (shown in FIG. 7) connected tothe device 700. For example, the driver 702 can send control signals716(A), 716(B), and 716(C) to the light-emitting devices 708(A), 708(B),and 708(C), respectively. The control signals can direct a singlelight-emitting device to be active at a time. For instance, the controlsignals can stagger activation of the light-emitting devices.

The following description provides an example of a sequential ToF imagecapturing process performed by the device 800 and the device 900. At atime T₁, the device 800 can orient the scanning mirror 814 atorientation O₁, causing light to be directed towards a first FOVportion. In response (e.g., at a time synchronized with time T₁), thesensor 910 can receive light corresponding to an illumination reflectionof the first FOV portion (e.g., a FOV portion corresponding to the ToFframe FOV_(1,1) of FIG. 6B). At a time T₂, the device 800 can orient thescanning mirror 814 at orientation O₂, causing light to be directedtowards a second FOV portion. In response (e.g., at a time synchronizedwith time T₂), the sensor 910 can receive light corresponding to anillumination reflection of the second FOV portion (e.g., a FOV portioncorresponding to the ToF frame FOV_(1,2) of FIG. 6B). At a time T₃, thedevice 800 can orient that scanning mirror at orientation O₃, causinglight to be directed towards a third FOV portion. In response (e.g., ata time synchronized with time T₃), the sensor 910 can receive lightcorresponding to an illumination reflection of the third FOV portion(e.g., a FOV portion corresponding to the ToF frame FOV_(1,3) of FIG.6B). In some cases, the amount of time between time T₁ and time T₂ (andbetween time T₂ and time T₃) can be sufficient for the device 800 toproject an illumination signal to one FOV, the device 900 to receive anillumination reflection corresponding to the illumination signal, andthe device 900 (and/or an additional processor or computing device) toprocess the illumination reflection for incorporation into a depth map.In some examples, the sequential illumination process can be directed atleast in part by a driver 802 communicatively connected to the device800. For example, the driver 802 can send a control signal 816 to thescanning mirror 814. The control signal 816 can direct the scanningmirror 814 to adjust the orientation of the scanning mirror 814 atappropriate times in order to scan each FOV portion.

In some cases, the device 700 can facilitate sequentially obtainingdistance measurements for a plurality of FOV portions without mechanicalmovement. For example, the device 700 can illuminate a desired FOVportion via selective activation of a light-emitting device, rather thanby adjusting the physical position and/or orientation of the componentsof the device 700. Eliminating the need for mechanical movement canreduce the risk of failures within the device 700. As described above,the device 800 can obtain distance measurements for a plurality of FOVportions via mechanical adjustment of a scanning mirror. However, themechanical fatigue (and corresponding performance degradation) caused bythe mechanical adjustment of the scanning mirror may be negligible.Further, the device 800 can include a single light-emitting device,which can reduce the size, cost, and/or complexity of the device.

Returning to FIG. 5, as mentioned above, the image processing system 500can synchronize the projection system 502 and the receiving system 508via a synchronization signal. For example, the image processing system500 can implement a synchronization scheme that includes sending a firstcontrol signal to the projection system 502 and sending a second controlsignal to the receiving system 508. In one example, the first controlsignal can indicate a sequential illumination order and the secondcontrol signal can indicate a corresponding sequential receiving order.In some cases, the first control signal and the second control signalcan be the same (or a similar) time-synchronized signal.

FIG. 10A is a diagram of an example synchronized ToF system 1000(A). Inone example, all or a portion of the synchronized ToF system 1000(A) cancorrespond to the device 700 of FIG. 7 and the device 900 of FIG. 9A. Asshown, the synchronized ToF system 1000(A) can include a controller1002. The controller 1002 can send a control signal 1004 to a driver ofthe device 700 (e.g., the driver 702). In some cases, the control signal1004 can represent a VCSEL-select signal. For example, the controlsignal 1004 can direct the driver 702 to selectively activate thelight-emitting devices 708 of the device 700 at appropriate times. Inone example, the control signal 1004 can direct the driver 702 to sendthe control signals 716(A), 716(B), and/or 716(C) shown in FIG. 7. Insome cases, the control signals 716(A), 716(B), and/or 716(C) caninclude a driving current (e.g., a pulse train drive current or acontinuous wave (CW) modulated drive current) suitable for operation ofthe light-emitting devices 708. The controller 1002 can also send acontrol signal 1006 to sensor electronics 1008 of the device 900. Thesensor electronics 1008 can include hardware and/or software componentsconfigured to determine distance measurements associated withillumination reflections. In some cases, the control signal 1006 can besynchronized with the control signal 1004. For instance, the controlsignal 1006 can indicate, to the sensor electronics 1008, which FOVportion is currently being illuminated and, therefore, which FOV portioncorresponds to the current illumination reflection. Based on the controlsignals 1004 and 1006, the synchronized ToF system 1000(A) can labeldistance measurements in association with corresponding FOV portions.

FIG. 10B is a diagram of an example synchronized ToF system 1000(B). Inone example, all or a portion of the synchronized ToF system 1000(B) cancorrespond to the device 800 of FIG. 8 and the device 900 of FIG. 9A. Asshown, the synchronized ToF system 1000(B) can include a controller1012. The controller 1012 can send a control signal 1014 to a driver ofthe device 800 (e.g., the driver 802). In some cases, the control signal1014 can represent an angle-select signal. For instance, the controlsignal 1014 can direct the driver 802 to adjust the angle of thescanning mirror 814 at appropriate times. In some examples, thecontroller 1012 can send a control signal 1020 to an additional driver1016. In one example, the driver 1016 can be a driver for thelight-emitting device 808 of the device 800. In some cases, the controlsignal 1020 can represent a VCSEL-synchronization signal. For example,the control signal 1020 can direct the driver 1016 to provide anappropriate driving current to the light-emitting device 808 in order tosequentially illuminate FOV portions. As shown, the controller 1012 cansend a control signal 1022 to sensor electronics 1018 of the device 900.The sensor electronics 1018 can include hardware and/or softwarecomponents configured to determine distance measurements associated withillumination reflections. In some cases, the control signal 1022 can besynchronized with the control signal 1014 and/or the control signal1020. For instance, the control signal 1022 can indicate, to the sensorelectronics 1018, which FOV portion is currently being illuminated and,therefore, which FOV portion corresponds to the current illuminationreflection. Based on the control signals 1014, 1020, and/or 1022, thesynchronized ToF system 1000(B) can label distance measurements inassociation with corresponding FOV portions.

Using the systems and techniques described herein, a ToF system cangenerate depth maps (e.g. depth image maps) having a resolution thatexceeds the full resolution of a ToF sensor of the ToF system. Thesystems and techniques also allow the ToF system to generatehigh-resolution depth maps without incurring substantial delays inprocessing time and without increasing the size of the sensor (thusdecreasing the cost of the sensor).

FIG. 11 is a flow diagram illustrating an example process 1100 for highresolution ToF depth imaging. For the sake of clarity, the process 1100is described with references to the systems and devices of FIG. 5, FIG.7, FIG. 8, and FIG. 9A. The steps or operations outlined herein areexamples and can be implemented in any combination thereof, includingcombinations that exclude, add, or modify certain steps or operations.

At operation 1102, the process 1100 includes illuminating at least oneportion of an FOV of a projection system including one or morelight-emitting devices, wherein the entire FOV includes a plurality ofFOV portions. In some examples, a first FOV portion of the plurality ofFOV portions can partially overlap at least a second FOV portion of theplurality of FOV portions. In some examples, illuminating the pluralityof FOV portions includes illuminating each of the plurality of FOVportions in a sequential illumination order (e.g., by illuminating asingle FOV portion at a time). In such examples, receiving eachillumination reflection of the plurality of illumination reflectionsincludes receiving each reflection in a sequential receiving order thatcorresponds to the sequential illumination order. The process 1100 caninclude generating each image of the plurality of images based on eachillumination reflection of the plurality of illumination reflections.Referring to FIG. 5 as an illustrative example, the projection system502 of the image processing system 500 can illuminate each of theplurality of FOV portions in the sequential illumination order at leastin part by illuminating a single FOV portion at a time. In someexamples, the projection system 502 can sequentially project theillumination signals 512 using the light-emitting device(s) 504 and/orthe light-directing device(s) 506.

In one illustrative example, the light-emitting device(s) 504 caninclude a plurality of light-emitting devices and the light-directingdevice(s) 506 can include a plurality of projection lenses. Eachprojection lens of the plurality of projection lenses can be configuredto project (e.g., together with a prism, such as prism 714) lightemitted by one of the plurality of light-emitting devices towards adifferent FOV portion of the plurality of FOV portions. In someexamples, the projection system 502 can include a segmented prism arrayconfigured to direct the light to each FOV portion of the plurality ofFOV portions. For instance, the plurality of light-emitting devices 504can include the light-emitting devices 708 of FIG. 7, and the pluralityof light-directing devices can include the prism 714 of FIG. 7. In oneexample, the prism 714 can be a segmented prism array that includesmultiple prism segments. Each prism segment of the prism 714 can bepositioned above one of the light-emitting devices 708. Further, eachprism segment of the prism 714 can be oriented at a different anglerelative to a plane of the light-emitting devices 708. In some examples,the projection system 502 can include one or more diffusers positionedrelative to the scanning mirror. The one or more diffusers areconfigured to diffuse the light emitted by the single light-emittingdevice. For instance, the device 700 of FIG. 7 (as an example of theprojection system 502) can include the diffuser array 712. As anotherexample, the device 800 of FIG. 8 can include the diffuser 812.

In another illustrative example, the light-emitting devices(s) 504 caninclude a single light-emitting device and the light-directing device(s)506 can include a scanning mirror. The scanning mirror can be configuredto project light emitted by the single light-emitting device towardsdifferent FOV portions of the plurality of FOV portions when oriented atdifferent orientations. For instance, the single light-emitting devicecan include the light-emitting device 808 of FIG. 8, and thesingle-light directing device can include the scanning mirror 814 ofFIG. 8. The scanning mirror 814 can be a scanning mirror. In some cases,the scanning mirror 814 can be positioned above the light-emittingdevice 808. Further, each different orientation of the scanning mirror814 can correspond to a different orientation angle between the scanningmirror 814 and a plane of the light-emitting device 808.

At operation 1104, the process 1100 includes sequentially capturing(e.g., by a sensor of a receiving system, such as receiving system 508)a plurality of images based on a plurality of illumination reflectionscorresponding to light emitted by the one or more light-emittingdevices. Each image of the plurality of images corresponds to one of theplurality of FOV portions. Further, an image resolution associated witheach image of the plurality of images corresponds to a full resolutionof the sensor. In one illustrative example referring to FIG. 5, thereceiving system 508 of the image processing system 500 can receive theillumination reflections 514 in a sequential receiving order thatcorresponds to the sequential illumination order of the illuminationsignals 512. A sensor of the receiving system 508 can capture an imagefor each illumination reflection. For example, the projection system 502can illuminate a first FOV portion of the plurality of FOV portions andthe receiving system 508 can receive a first illumination reflection andthe sensor can capture a first image corresponding to the first FOVportion. After receiving the first illumination reflection, theprojection system 502 can illuminate a second FOV portion of theplurality of FOV portions. The receiving system 508 can then receive asecond illumination reflection and the sensor can capture a second imagecorresponding to the second FOV portion.

In one example, the projection system 502 and the receiving system 508can be synchronized. For example, the processor 518 of the imageprocessing system 500 can send, to the projection system 502, a firstcontrol signal directing the projection system 502 to illuminate aparticular FOV portion of the plurality of FOV portions. 506. Theprocessor 518 can also send, to the receiving system 508, a secondcontrol signal directing the receiving system 508 to associate anillumination reflection received by the sensor 510 with the particularFOV portion. The first control signal and the second control signal canbe time-synchronized.

In some examples, the receiving system 508 can include an array of imagelenses, with each image lens of the array being configured to project,to a sensor of the receiving system 508, light associated with adifferent portion of the scene corresponding to a respective FOV portionof the plurality of FOV portions. For instance, the device 900 of FIG.9A (as an example of the receiving system 508) can include the imagelens array 902. In some cases, the receiving system 508 can include afilter positioned above the sensor 510. For instance, the device 900 caninclude the filter 922. The filter 922 can be configured to transmitlight with a frequency corresponding to a frequency of light emitted bythe light-emitting device(s) 504 (e.g., the light-emitting devices 708of FIG. 7 and/or the light-emitting device 808 of FIG. 8).

At operation 1106, the process 1100 includes generating, using theplurality of images (and/or illumination reflections), an increasedresolution depth map associated with the entire FOV. For instance, theprocessor 518 can generate the depth map 516, which can include anincreased resolution depth map based on the various full resolutionimages associated with each FOV portion of the entire FOV. In oneexample, the processor 518 can generate a plurality of partial distancemeasurements (e.g., as images) that each correspond to one of theillumination reflections 514. The processor 518 can then generate thedepth map 516 by combining the plurality of partial distancemeasurements. Further, in one example, the image resolution of the depthmap 516 can correspond to a maximum resolution of the sensor 510multiplied by the number of individual FOV portions.

In some examples, the process 1100 and/or other processes describedherein may be performed by one or more computing devices or apparatuses.In some examples, the process 1100 and/or other processes describedherein can be performed by the image processing system 500 shown in FIG.5, the device 700 shown in FIG. 7, the device 800 shown in FIG. 8, thedevice 900 shown in FIG. 9A, the synchronized ToF system 1000(A) shownin FIG. 10A, the synchronized ToF system 1000(B) shown in FIG. 10B,and/or one or more computing devices with the computing devicearchitecture 1100 shown in FIG. 1100. In some cases, such a computingdevice or apparatus may include a processor, microprocessor,microcomputer, or other component of a device that is configured tocarry out the steps of the process 1100. In some examples, suchcomputing device or apparatus may include one or more sensors configuredto capture image data. For example, the computing device can include asmartphone, a camera, a head-mounted display, a mobile device, or othersuitable device. In some examples, such computing device or apparatusmay include a camera configured to capture one or more images or videos.In some cases, such computing device may include a display fordisplaying images. In some examples, the one or more sensors and/orcamera are separate from the computing device, in which case thecomputing device receives the sensed data. Such computing device mayfurther include a network interface configured to communicate data.

The components of the computing device can be implemented in circuitry.For example, the components can include and/or can be implemented usingelectronic circuits or other electronic hardware, which can include oneor more programmable electronic circuits (e.g., microprocessors,graphics processing units (GPUs), digital signal processors (DSPs),central processing units (CPUs), and/or other suitable electroniccircuits), and/or can include and/or be implemented using computersoftware, firmware, or any combination thereof, to perform the variousoperations described herein. The computing device may further include adisplay (as an example of the output device or in addition to the outputdevice), a network interface configured to communicate and/or receivethe data, any combination thereof, and/or other component(s). Thenetwork interface may be configured to communicate and/or receiveInternet Protocol (IP) based data or other type of data.

The process 1100 is illustrated as a logical flow diagram, theoperations of which represent sequences of operations that can beimplemented in hardware, computer instructions, or a combinationthereof. In the context of computer instructions, the operationsrepresent computer-executable instructions stored on one or morecomputer-readable storage media that, when executed by one or moreprocessors, perform the recited operations. Generally,computer-executable instructions include routines, programs, objects,components, data structures, and the like that perform particularfunctions or implement particular data types. The order in which theoperations are described is not intended to be construed as alimitation, and any number of the described operations can be combinedin any order and/or in parallel to implement the processes.

Additionally, the process 1100 and/or other processes described hereinmay be performed under the control of one or more computer systemsconfigured with executable instructions and may be implemented as code(e.g., executable instructions, one or more computer programs, or one ormore applications) executing collectively on one or more processors, byhardware, or combinations thereof. As noted above, the code may bestored on a computer-readable or machine-readable storage medium, forexample, in the form of a computer program comprising a plurality ofinstructions executable by one or more processors. The computer-readableor machine-readable storage medium may be non-transitory.

FIG. 12 is a diagram illustrating an example of a system forimplementing certain aspects of the present technology. In particular,FIG. 12 illustrates an example of computing system 1200, which can befor example any computing device making up internal computing system, aremote computing system, a camera, or any component thereof in which thecomponents of the system are in communication with each other usingconnection 1205. Connection 1205 can be a physical connection using abus, or a direct connection into processor 1210, such as in a chipsetarchitecture. Connection 1205 can also be a virtual connection,networked connection, or logical connection.

In some examples, computing system 1200 is a distributed system in whichthe functions described in this disclosure can be distributed within adatacenter, multiple data centers, a peer network, etc. In someexamples, one or more of the described system components represents manysuch components each performing some or all of the function for whichthe component is described. In some cases, the components can bephysical or virtual devices.

Example system 1200 includes at least one processing unit (CPU orprocessor) 1210 and connection 1205 that couples various systemcomponents including system memory 1215, such as read-only memory (ROM)1220 and random access memory (RAM) 1225 to processor 1210. Computingsystem 1200 can include a cache 1212 of high-speed memory connecteddirectly with, in close proximity to, or integrated as part of processor1210.

Processor 1210 can include any general purpose processor and a hardwareservice or software service, such as services 1232, 1234, and 1236stored in storage device 1230, configured to control processor 1210 aswell as a special-purpose processor where software instructions areincorporated into the actual processor design. Processor 1210 mayessentially be a completely self-contained computing system, containingmultiple cores or processors, a bus, memory controller, cache, etc. Amulti-core processor may be symmetric or asymmetric.

To enable user interaction, computing system 1200 includes an inputdevice 1245, which can represent any number of input mechanisms, such asa microphone for speech, a touch-sensitive screen for gesture orgraphical input, keyboard, mouse, motion input, speech, etc. Computingsystem 1200 can also include output device 1235, which can be one ormore of a number of output mechanisms. In some instances, multimodalsystems can enable a user to provide multiple types of input/output tocommunicate with computing system 1200. Computing system 1200 caninclude communications interface 1240, which can generally govern andmanage the user input and system output. The communication interface mayperform or facilitate receipt and/or transmission wired or wirelesscommunications using wired and/or wireless transceivers, including thosemaking use of an audio jack/plug, a microphone jack/plug, a universalserial bus (USB) port/plug, an Apple® Lightning® port/plug, an Ethernetport/plug, a fiber optic port/plug, a proprietary wired port/plug, aBLUETOOTH® wireless signal transfer, a BLUETOOTH® low energy (BLE)wireless signal transfer, an IBEACON® wireless signal transfer, aradio-frequency identification (RFID) wireless signal transfer,near-field communications (NFC) wireless signal transfer, dedicatedshort range communication (DSRC) wireless signal transfer, 802.12 Wi-Fiwireless signal transfer, wireless local area network (WLAN) signaltransfer, Visible Light Communication (VLC), Worldwide Interoperabilityfor Microwave Access (WiMAX), Infrared (IR) communication wirelesssignal transfer, Public Switched Telephone Network (PSTN) signaltransfer, Integrated Services Digital Network (ISDN) signal transfer,3G/4G/5G/LTE cellular data network wireless signal transfer, ad-hocnetwork signal transfer, radio wave signal transfer, microwave signaltransfer, infrared signal transfer, visible light signal transfer,ultraviolet light signal transfer, wireless signal transfer along theelectromagnetic spectrum, or some combination thereof. Thecommunications interface 1240 may also include one or more GlobalNavigation Satellite System (GNSS) receivers or transceivers that areused to determine a location of the computing system 1200 based onreceipt of one or more signals from one or more satellites associatedwith one or more GNSS systems. GNSS systems include, but are not limitedto, the US-based Global Positioning System (GPS), the Russia-basedGlobal Navigation Satellite System (GLONASS), the China-based BeiDouNavigation Satellite System (BDS), and the Europe-based Galileo GNSS.There is no restriction on operating on any particular hardwarearrangement, and therefore the basic features here may easily besubstituted for improved hardware or firmware arrangements as they aredeveloped.

Storage device 1230 can be a non-volatile and/or non-transitory and/orcomputer-readable memory device and can be a hard disk or other types ofcomputer readable media which can store data that are accessible by acomputer, such as magnetic cassettes, flash memory cards, solid statememory devices, digital versatile disks, cartridges, a floppy disk, aflexible disk, a hard disk, magnetic tape, a magnetic strip/stripe, anyother magnetic storage medium, flash memory, memristor memory, any othersolid-state memory, a compact disc read only memory (CD-ROM) opticaldisc, a rewritable compact disc (CD) optical disc, digital video disk(DVD) optical disc, a blu-ray disc (BDD) optical disc, a holographicoptical disk, another optical medium, a secure digital (SD) card, amicro secure digital (microSD) card, a Memory Stick® card, a smartcardchip, a EMV chip, a subscriber identity module (SIM) card, amini/micro/nano/pico SIM card, another integrated circuit (IC)chip/card, random access memory (RAM), static RAM (SRAM), dynamic RAM(DRAM), read-only memory (ROM), programmable read-only memory (PROM),erasable programmable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cachememory (L1/L2/L3/L4/L5/L#), resistive random-access memory (RRAM/ReRAM),phase change memory (PCM), spin transfer torque RAM (STT-RAM), anothermemory chip or cartridge, and/or a combination thereof.

The storage device 1230 can include software services, servers,services, etc., that when the code that defines such software isexecuted by the processor 1210, it causes the system to perform afunction. In some examples, a hardware service that performs aparticular function can include the software component stored in acomputer-readable medium in connection with the necessary hardwarecomponents, such as processor 1210, connection 1205, output device 1235,etc., to carry out the function.

As used herein, the term “computer-readable medium” includes, but is notlimited to, portable or non-portable storage devices, optical storagedevices, and various other mediums capable of storing, containing, orcarrying instruction(s) and/or data. A computer-readable medium mayinclude a non-transitory medium in which data can be stored and thatdoes not include carrier waves and/or transitory electronic signalspropagating wirelessly or over wired connections. Examples of anon-transitory medium may include, but are not limited to, a magneticdisk or tape, optical storage media such as compact disk (CD) or digitalversatile disk (DVD), flash memory, memory or memory devices. Acomputer-readable medium may have stored thereon code and/ormachine-executable instructions that may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a class, or any combination of instructions, datastructures, or program statements. A code segment may be coupled toanother code segment or a hardware circuit by passing and/or receivinginformation, data, arguments, parameters, or memory contents.Information, arguments, parameters, data, etc. may be passed, forwarded,or transmitted using any suitable means including memory sharing,message passing, token passing, network transmission, or the like.

In some examples, the computer-readable storage devices, mediums, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, non-transitory computer-readablestorage media expressly exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

Specific details are provided in the description above to provide athorough understanding of the examples provided herein. However, it willbe understood by one of ordinary skill in the art that the examples maybe practiced without these specific details. For clarity of explanation,in some instances the present technology may be presented as includingindividual functional blocks including functional blocks comprisingdevices, device components, steps or routines in a method embodied insoftware, or combinations of hardware and software. Additionalcomponents may be used other than those shown in the figures and/ordescribed herein. For example, circuits, systems, networks, processes,and other components may be shown as components in block diagram form inorder not to obscure the examples in unnecessary detail. In otherinstances, well-known circuits, processes, algorithms, structures, andtechniques may be shown without unnecessary detail in order to avoidobscuring the examples.

Individual examples may be described above as a process or method whichis depicted as a flowchart, a flow diagram, a data flow diagram, astructure diagram, or a block diagram. Although a flowchart may describethe operations as a sequential process, many of the operations can beperformed in parallel or concurrently. In addition, the order of theoperations may be re-arranged. A process is terminated when itsoperations are completed, but could have additional steps not includedin a figure. A process may correspond to a method, a function, aprocedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination can correspond to a return of thefunction to the calling function or the main function.

Processes and methods according to the above-described examples can beimplemented using computer-executable instructions that are stored orotherwise available from computer-readable media. Such instructions caninclude, for example, instructions and data which cause or otherwiseconfigure a general purpose computer, special purpose computer, or aprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The computer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware,source code, etc. Examples of computer-readable media that may be usedto store instructions, information used, and/or information createdduring methods according to described examples include magnetic oroptical disks, flash memory, USB devices provided with non-volatilememory, networked storage devices, and so on.

Devices implementing processes and methods according to thesedisclosures can include hardware, software, firmware, middleware,microcode, hardware description languages, or any combination thereof,and can take any of a variety of form factors. When implemented insoftware, firmware, middleware, or microcode, the program code or codesegments to perform the necessary tasks (e.g., a computer-programproduct) may be stored in a computer-readable or machine-readablemedium. A processor(s) may perform the necessary tasks. Typical examplesof form factors include laptops, smart phones, mobile phones, tabletdevices or other small form factor personal computers, personal digitalassistants, rackmount devices, standalone devices, and so on.Functionality described herein also can be embodied in peripherals oradd-in cards. Such functionality can also be implemented on a circuitboard among different chips or different processes executing in a singledevice, by way of further example.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are example means for providing the functionsdescribed in the disclosure.

In the foregoing description, aspects of the application are describedwith reference to specific examples thereof, but those skilled in theart will recognize that the application is not limited thereto. Thus,while illustrative examples of the application have been described indetail herein, it is to be understood that the inventive concepts may beotherwise variously embodied and employed, and that the appended claimsare intended to be construed to include such variations, except aslimited by the prior art. Various features and aspects of theabove-described application may be used individually or jointly.Further, examples can be utilized in any number of environments andapplications beyond those described herein without departing from thebroader spirit and scope of the specification. The specification anddrawings are, accordingly, to be regarded as illustrative rather thanrestrictive. For the purposes of illustration, methods were described ina particular order. It should be appreciated that in alternate examples,the methods may be performed in a different order than that described.

One of ordinary skill will appreciate that the less than (“<”) andgreater than (“>”) symbols or terminology used herein can be replacedwith less than or equal to (“≤”) and greater than or equal to (“≥”)symbols, respectively, without departing from the scope of thisdescription.

Where components are described as being “configured to” perform certainoperations, such configuration can be accomplished, for example, bydesigning electronic circuits or other hardware to perform theoperation, by programming programmable electronic circuits (e.g.,microprocessors, or other suitable electronic circuits) to perform theoperation, or any combination thereof.

The phrase “coupled to” refers to any component that is physicallyconnected to another component either directly or indirectly, and/or anycomponent that is in communication with another component (e.g.,connected to the other component over a wired or wireless connection,and/or other suitable communication interface) either directly orindirectly.

Claim language or other language reciting “at least one of” a set and/or“one or more” of a set indicates that one member of the set or multiplemembers of the set (in any combination) satisfy the claim. For example,claim language reciting “at least one of A and B” means A, B, or A andB. In another example, claim language reciting “at least one of A, B,and C” means A, B, C, or A and B, or A and C, or B and C, or A and B andC. The language “at least one of” a set and/or “one or more” of a setdoes not limit the set to the items listed in the set. For example,claim language reciting “at least one of A and B” can mean A, B, or Aand B, and can additionally include items not listed in the set of A andB.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the examples disclosedherein may be implemented as electronic hardware, computer software,firmware, or combinations thereof. To clearly illustrate thisinterchangeability of hardware and software, various illustrativecomponents, blocks, modules, circuits, and steps have been describedabove generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled artisans may implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the present application.

The techniques described herein may also be implemented in electronichardware, computer software, firmware, or any combination thereof. Suchtechniques may be implemented in any of a variety of devices such asgeneral purposes computers, wireless communication device handsets, orintegrated circuit devices having multiple uses including application inwireless communication device handsets and other devices. Any featuresdescribed as modules or components may be implemented together in anintegrated logic device or separately as discrete but interoperablelogic devices. If implemented in software, the techniques may berealized at least in part by a computer-readable data storage mediumcomprising program code including instructions that, when executed,performs one or more of the methods described above. Thecomputer-readable data storage medium may form part of a computerprogram product, which may include packaging materials. Thecomputer-readable medium may comprise memory or data storage media, suchas random access memory (RAM) such as synchronous dynamic random accessmemory (SDRAM), read-only memory (ROM), non-volatile random accessmemory (NVRAM), electrically erasable programmable read-only memory(EEPROM), FLASH memory, magnetic or optical data storage media, and thelike. The techniques additionally, or alternatively, may be realized atleast in part by a computer-readable communication medium that carriesor communicates program code in the form of instructions or datastructures and that can be accessed, read, and/or executed by acomputer, such as propagated signals or waves.

The program code may be executed by a processor, which may include oneor more processors, such as one or more digital signal processors(DSPs), general purpose microprocessors, an application specificintegrated circuits (ASICs), field programmable logic arrays (FPGAs), orother equivalent integrated or discrete logic circuitry. Such aprocessor may be configured to perform any of the techniques describedin this disclosure. A general purpose processor may be a microprocessor;but in the alternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration. Accordingly, the term “processor,” as used herein mayrefer to any of the foregoing structure, any combination of theforegoing structure, or any other structure or apparatus suitable forimplementation of the techniques described herein. In addition, in someaspects, the functionality described herein may be provided withindedicated software modules or hardware modules configured for encodingand decoding, or incorporated in a combined video encoder-decoder(CODEC).

Illustrative aspects of the present disclosure include:

Aspect 1: An apparatus for high resolution time-of-flight depth imaging,the apparatus comprising: a projection system including one or morelight-emitting devices, each light-emitting device of the one or morelight-emitting devices configured to illuminate at least one portion ofan entire field-of-view (FOV) of the projection system, wherein theentire FOV includes a plurality of FOV portions; a receiving systemincluding a sensor configured to sequentially capture a plurality ofimages based on a plurality of illumination reflections corresponding tolight emitted by the one or more light-emitting devices, wherein eachimage of the plurality of images corresponds to one of the plurality ofFOV portions, and wherein an image resolution associated with each imageof the plurality of images corresponds to a full resolution of thesensor; and a processor configured to generate, using the plurality ofimages, an increased resolution depth map associated with the entireFOV.

Aspect 2: The apparatus of claim 1, wherein: the projection system isconfigured to illuminate each of the plurality of FOV portions in asequential illumination order, the sequential illumination orderincluding illuminating a single FOV portion at a time; and the receivingsystem is configured to receive each illumination reflection of theplurality of illumination reflections in a sequential receiving orderthat corresponds to the sequential illumination order and to generateeach image of the plurality of images based on each illuminationreflection.

Aspect 3: The apparatus of claim 2, wherein, to illuminate each of theplurality of FOV portions in the sequential illumination order, theprojection system is configured to: illuminate a first FOV portion ofthe plurality of FOV portions; receive a first illumination reflectioncorresponding to the first FOV portion; after receiving the firstillumination reflection, illuminate a second FOV portion of theplurality of FOV portions; and receive a second illumination reflectioncorresponding to the second FOV portion.

Aspect 4: The apparatus of claim 1, wherein: the one or morelight-emitting devices include a plurality of light-emitting devices;and the projection system includes a plurality of projection lenses,each projection lens of the plurality of projection lenses beingconfigured to project light emitted by one of the plurality oflight-emitting devices towards a different FOV portion of the pluralityof FOV portions.

Aspect 5: The apparatus of claim 4, wherein the projection systemincludes one or more diffusers positioned relative to the plurality ofprojection lenses, the one or more diffusers being configured to diffusethe light emitted by the plurality of light-emitting devices.

Aspect 6: The apparatus of claim 4, wherein each of the plurality ofprojection lenses is positioned above the plurality of light-emittingdevices.

Aspect 7: The apparatus of claim 1, wherein the projection systemincludes a segmented prism array, the segmented prism array beingconfigured to direct the light to each FOV portion of the plurality ofFOV portions.

Aspect 8: The apparatus of claim 1, wherein: the one or morelight-emitting devices include a single light-emitting device; and theprojection system includes a scanning mirror, wherein the scanningmirror is configured to project light emitted by the singlelight-emitting device towards different FOV portions of the plurality ofFOV portions when oriented at different orientations.

Aspect 9: The apparatus of claim 8, wherein the scanning mirror includesa micro electro mechanical system (MEMS) mirror.

Aspect 10: The apparatus of claim 8, wherein: the scanning mirror ispositioned above the single light-emitting device; and each of thedifferent orientations of the scanning mirror corresponds to a differentorientation angle between the scanning mirror and a plane of the singlelight-emitting device.

Aspect 11: The apparatus of claim 8, wherein the projection systemincludes one or more diffusers positioned relative to the scanningmirror, the one or more diffusers being configured to diffuse the lightemitted by the single light-emitting device.

Aspect 12: The apparatus of claim 1, wherein the receiving systemincludes an array of image lenses, each image lens of the array of imagelenses being configured to project, to the sensor, light associated witha different portion of the scene corresponding to a respective FOVportion of the plurality of FOV portions.

Aspect 13: The apparatus of claim 1, wherein the receiving systemfurther comprises a filter positioned above the sensor, the filter beingconfigured to transmit light with a frequency corresponding to afrequency of light emitted by the one or more light-emitting devices.

Aspect 14: The apparatus of claim 1, wherein the processor is configuredto synchronize the projection system and the receiving system based atleast in part on: sending, to the projection system, a first controlsignal directing the projection system to illuminate a particular FOVportion of the plurality of FOV portions; and sending, to the receivingsystem, a second control signal directing the receiving system toassociate an illumination reflection received by the sensor with theparticular FOV portion, wherein the first control signal and the secondcontrol signal are time-synchronized.

Aspect 15: The apparatus of claim 1, wherein a first FOV portion of theplurality of FOV portions partially overlaps at least a second FOVportion of the plurality of FOV portions.

Aspect 16: The apparatus of claim 1, wherein, to generate the increasedresolution depth map associated with the entire FOV, the processor isconfigured to: generate a plurality of partial distance measurements,each of the plurality of partial distance measurements corresponding toone of the plurality of illumination reflections; and combine theplurality of partial distance measurements.

Aspect 17: The apparatus of claim 1, wherein an image resolution of theincreased resolution depth map corresponds to a maximum resolution ofthe sensor multiplied by a number of the plurality of FOV portions.

Aspect 18: A method for high resolution time-of-flight depth imaging,the method comprising: illuminating, using one or more light-emittingdevices of a projection system, a plurality of field-of-view (FOV)portions of an entire FOV of the projection system, wherein eachlight-emitting device of the one or more light-emitting devices isconfigured to illuminate at least one portion of the entire FOV;sequentially capturing, by a sensor of a receiving system, a pluralityof images based on a plurality of illumination reflections correspondingto light emitted by the one or more light-emitting devices, wherein eachimage of the plurality of images corresponds to one of the plurality ofFOV portions, and wherein an image resolution associated with each imageof the plurality of images corresponds to a full resolution of thesensor; and generating, using the plurality of images, an increasedresolution depth map associated with the entire FOV.

Aspect 19: The method of claim 18, wherein: illuminating the pluralityof FOV portions includes illuminating each of the plurality of FOVportions in a sequential illumination order, the sequential illuminationorder including illuminating a single FOV portion at a time; receivingeach illumination reflection of the plurality of illuminationreflections includes receiving each reflection in a sequential receivingorder that corresponds to the sequential illumination order; and furthercomprising generating each image of the plurality of images based oneach illumination reflection of the plurality of illuminationreflections.

Aspect 20: The method of claim 19, wherein illuminating each of theplurality of FOV portions in the sequential illumination order includes:illuminating a first FOV portion of the plurality of FOV portions;receiving a first illumination reflection corresponding to the first FOVportion; after receiving the first illumination reflection, illuminatinga second FOV portion of the plurality of FOV portions; and receiving asecond illumination reflection corresponding to the second FOV portion.

Aspect 21: The method of claim 18, wherein: the one or morelight-emitting devices include a plurality of light-emitting devices;and the projection system includes a plurality of projection lenses,each projection lens of the plurality of projection lenses beingconfigured to project light emitted by one of the plurality oflight-emitting devices towards a different FOV portion of the pluralityof FOV portions.

Aspect 22: The method of claim 21, wherein the projection systemincludes one or more diffusers positioned relative to the plurality ofprojection lenses, the one or more diffusers being configured to diffusethe light emitted by the plurality of light-emitting devices.

Aspect 23: The method of claim 21, wherein each of the plurality oflenses is positioned above the plurality of light-emitting devices.

Aspect 24: The method of claim 18, wherein the projection systemincludes a segmented prism array, the segmented prism array beingconfigured to direct the light to each FOV portion of the plurality ofFOV portions.

Aspect 25: The method of claim 18, wherein: the one or morelight-emitting devices include a single light-emitting device; and theprojection system includes a scanning mirror, wherein the scanningmirror is configured to project light emitted by the singlelight-emitting device towards different FOV portions of the plurality ofFOV portions when oriented at different orientations.

Aspect 26: The method of claim 25, wherein the scanning mirror includesa micro electro mechanical system (MEMS) mirror.

Aspect 27: The method of claim 25, wherein: the scanning mirror ispositioned above the single light-emitting device; and each of thedifferent orientations of the scanning mirror corresponds to a differentorientation angle between the scanning mirror and a plane of the singlelight-emitting device.

Aspect 28: The apparatus of claim 25, wherein the projection systemincludes one or more diffusers positioned relative to the scanningmirror, the one or more diffusers being configured to diffuse the lightemitted by the single light-emitting device.

Aspect 29: The method of claim 18, wherein the receiving system includesan array of image lenses, each image lens of the array of image lensesbeing configured to project, to the sensor, light associated with adifferent portion of the scene corresponding to a respective FOV portionof the plurality of FOV portions.

Aspect 30: The method of claim 18, further comprising a filterpositioned above the sensor, the filter being configured to transmitlight with a frequency corresponding to a frequency of light emitted bythe one or more light-emitting devices.

Aspect 31: The method of claim 18, further comprising synchronizing theprojection system and the receiving system based at least in part on:sending, to the projection system, a first control signal directing theprojection system to illuminate a particular FOV portion of theplurality of FOV portions; and sending, to the receiving system, asecond control signal directing the receiving system to associate anillumination reflection received by the sensor with the particular FOVportion, wherein the first control signal and the second control signalare time-synchronized.

Aspect 32: The method of claim 18, wherein a first FOV portion of theplurality of FOV portions partially overlaps at least a second FOVportion of the plurality of FOV portions.

Aspect 33: The method of claim 18, wherein generating the increasedresolution depth map associated with the entire FOV includes: generatinga plurality of partial distance measurements, each of the plurality ofpartial distance measurements corresponding to one of the plurality ofillumination reflections; and combining the plurality of partialdistance measurements.

Aspect 34: The method of claim 18, wherein an image resolution of theincreased resolution depth map corresponds to a maximum resolution ofthe sensor multiplied by a number of the plurality of FOV portions.

Aspect 35: A non-transitory computer-readable storage medium havinginstructions stored therein which, when executed by one or moreprocessors, cause the one or more processors to perform operationsaccording to any of aspects 1 to 34.

Aspect 36: An apparatus comprising means for performing operationsaccording to any of aspects 1 to 34.

What is claimed is:
 1. An apparatus for high resolution time-of-flightdepth imaging, the apparatus comprising: a projection system includingone or more light-emitting devices, each light-emitting device of theone or more light-emitting devices configured to illuminate at least oneportion of an entire field-of-view (FOV) of the projection system,wherein the entire FOV includes a plurality of FOV portions; a receivingsystem including a sensor configured to sequentially capture a pluralityof images based on a plurality of illumination reflections correspondingto light emitted by the one or more light-emitting devices, wherein eachimage of the plurality of images corresponds to one of the plurality ofFOV portions, and wherein an image resolution associated with each imageof the plurality of images corresponds to a full resolution of thesensor; and a processor configured to generate, using the plurality ofimages, an increased resolution depth map associated with the entireFOV.
 2. The apparatus of claim 1, wherein: the projection system isconfigured to illuminate each of the plurality of FOV portions in asequential illumination order, the sequential illumination orderincluding illuminating a single FOV portion at a time; and the receivingsystem is configured to receive each illumination reflection of theplurality of illumination reflections in a sequential receiving orderthat corresponds to the sequential illumination order and to generateeach image of the plurality of images based on each illuminationreflection.
 3. The apparatus of claim 2, wherein, to illuminate each ofthe plurality of FOV portions in the sequential illumination order, theprojection system is configured to: illuminate a first FOV portion ofthe plurality of FOV portions; receive a first illumination reflectioncorresponding to the first FOV portion; after receiving the firstillumination reflection, illuminate a second FOV portion of theplurality of FOV portions; and receive a second illumination reflectioncorresponding to the second FOV portion.
 4. The apparatus of claim 1,wherein: the one or more light-emitting devices include a plurality oflight-emitting devices; and the projection system includes a pluralityof projection lenses, each projection lens of the plurality ofprojection lenses being configured to project light emitted by one ofthe plurality of light-emitting devices towards a different FOV portionof the plurality of FOV portions.
 5. The apparatus of claim 4, whereinthe projection system includes one or more diffusers positioned relativeto the plurality of projection lenses, the one or more diffusers beingconfigured to diffuse the light emitted by the plurality oflight-emitting devices.
 6. The apparatus of claim 4, wherein each of theplurality of projection lenses is positioned above the plurality oflight-emitting devices.
 7. The apparatus of claim 1, wherein theprojection system includes a segmented prism array, the segmented prismarray being configured to direct the light to each FOV portion of theplurality of FOV portions.
 8. The apparatus of claim 1, wherein: the oneor more light-emitting devices include a single light-emitting device;and the projection system includes a scanning mirror, wherein thescanning mirror is configured to project light emitted by the singlelight-emitting device towards different FOV portions of the plurality ofFOV portions when oriented at different orientations.
 9. The apparatusof claim 8, wherein the scanning mirror includes a micro electromechanical system (MEMS) mirror.
 10. The apparatus of claim 8, wherein:the scanning mirror is positioned above the single light-emittingdevice; and each of the different orientations of the scanning mirrorcorresponds to a different orientation angle between the scanning mirrorand a plane of the single light-emitting device.
 11. The apparatus ofclaim 8, wherein the projection system includes one or more diffuserspositioned relative to the scanning mirror, the one or more diffusersbeing configured to diffuse the light emitted by the singlelight-emitting device.
 12. The apparatus of claim 1, wherein thereceiving system includes an array of image lenses, each image lens ofthe array of image lenses being configured to project, to the sensor,light associated with a different portion of the scene corresponding toa respective FOV portion of the plurality of FOV portions.
 13. Theapparatus of claim 1, wherein the receiving system further comprises afilter positioned above the sensor, the filter being configured totransmit light with a frequency corresponding to a frequency of lightemitted by the one or more light-emitting devices.
 14. The apparatus ofclaim 1, wherein the processor is configured to synchronize theprojection system and the receiving system based at least in part on:sending, to the projection system, a first control signal directing theprojection system to illuminate a particular FOV portion of theplurality of FOV portions; and sending, to the receiving system, asecond control signal directing the receiving system to associate anillumination reflection received by the sensor with the particular FOVportion, wherein the first control signal and the second control signalare time-synchronized.
 15. The apparatus of claim 1, wherein a first FOVportion of the plurality of FOV portions partially overlaps at least asecond FOV portion of the plurality of FOV portions.
 16. The apparatusof claim 1, wherein, to generate the increased resolution depth mapassociated with the entire FOV, the processor is configured to: generatea plurality of partial distance measurements, each of the plurality ofpartial distance measurements corresponding to one of the plurality ofillumination reflections; and combine the plurality of partial distancemeasurements.
 17. The apparatus of claim 1, wherein an image resolutionof the increased resolution depth map corresponds to a maximumresolution of the sensor multiplied by a number of the plurality of FOVportions.
 18. A method for high resolution time-of-flight depth imaging,the method comprising: illuminating, using one or more light-emittingdevices of a projection system, a plurality of field-of-view (FOV)portions of an entire FOV of the projection system, wherein eachlight-emitting device of the one or more light-emitting devices isconfigured to illuminate at least one portion of the entire FOV;sequentially capturing, by a sensor of a receiving system, a pluralityof images based on a plurality of illumination reflections correspondingto light emitted by the one or more light-emitting devices, wherein eachimage of the plurality of images corresponds to one of the plurality ofFOV portions, and wherein an image resolution associated with each imageof the plurality of images corresponds to a full resolution of thesensor; and generating, using the plurality of images, an increasedresolution depth map associated with the entire FOV.
 19. The method ofclaim 18, wherein: illuminating the plurality of FOV portions includesilluminating each of the plurality of FOV portions in a sequentialillumination order, the sequential illumination order includingilluminating a single FOV portion at a time; receiving each illuminationreflection of the plurality of illumination reflections includesreceiving each reflection in a sequential receiving order thatcorresponds to the sequential illumination order; and further comprisinggenerating each image of the plurality of images based on eachillumination reflection of the plurality of illumination reflections.20. The method of claim 19, wherein illuminating each of the pluralityof FOV portions in the sequential illumination order includes:illuminating a first FOV portion of the plurality of FOV portions;receiving a first illumination reflection corresponding to the first FOVportion; after receiving the first illumination reflection, illuminatinga second FOV portion of the plurality of FOV portions; and receiving asecond illumination reflection corresponding to the second FOV portion.21. The method of claim 18, wherein: the one or more light-emittingdevices include a plurality of light-emitting devices; and theprojection system includes a plurality of projection lenses, eachprojection lens of the plurality of projection lenses being configuredto project light emitted by one of the plurality of light-emittingdevices towards a different FOV portion of the plurality of FOVportions.
 22. The method of claim 21, wherein the projection systemincludes one or more diffusers positioned relative to the plurality ofprojection lenses, the one or more diffusers being configured to diffusethe light emitted by the plurality of light-emitting devices.
 23. Themethod of claim 21, wherein each of the plurality of lenses ispositioned above the plurality of light-emitting devices.
 24. The methodof claim 18, wherein the projection system includes a segmented prismarray, the segmented prism array being configured to direct the light toeach FOV portion of the plurality of FOV portions.
 25. The method ofclaim 18, wherein: the one or more light-emitting devices include asingle light-emitting device; and the projection system includes ascanning mirror, wherein the scanning mirror is configured to projectlight emitted by the single light-emitting device towards different FOVportions of the plurality of FOV portions when oriented at differentorientations.
 26. The method of claim 25, wherein the scanning mirrorincludes a micro electro mechanical system (MEMS) mirror.
 27. The methodof claim 25, wherein: the scanning mirror is positioned above the singlelight-emitting device; and each of the different orientations of thescanning mirror corresponds to a different orientation angle between thescanning mirror and a plane of the single light-emitting device.
 28. Theapparatus of claim 25, wherein the projection system includes one ormore diffusers positioned relative to the scanning mirror, the one ormore diffusers being configured to diffuse the light emitted by thesingle light-emitting device.
 29. The method of claim 18, wherein thereceiving system includes an array of image lenses, each image lens ofthe array of image lenses being configured to project, to the sensor,light associated with a different portion of the scene corresponding toa respective FOV portion of the plurality of FOV portions.
 30. Themethod of claim 18, further comprising a filter positioned above thesensor, the filter being configured to transmit light with a frequencycorresponding to a frequency of light emitted by the one or morelight-emitting devices.
 31. The method of claim 18, further comprisingsynchronizing the projection system and the receiving system based atleast in part on: sending, to the projection system, a first controlsignal directing the projection system to illuminate a particular FOVportion of the plurality of FOV portions; and sending, to the receivingsystem, a second control signal directing the receiving system toassociate an illumination reflection received by the sensor with theparticular FOV portion, wherein the first control signal and the secondcontrol signal are time-synchronized.
 32. The method of claim 18,wherein a first FOV portion of the plurality of FOV portions partiallyoverlaps at least a second FOV portion of the plurality of FOV portions.33. The method of claim 18, wherein generating the increased resolutiondepth map associated with the entire FOV includes: generating aplurality of partial distance measurements, each of the plurality ofpartial distance measurements corresponding to one of the plurality ofillumination reflections; and combining the plurality of partialdistance measurements.
 34. The method of claim 18, wherein an imageresolution of the increased resolution depth map corresponds to amaximum resolution of the sensor multiplied by a number of the pluralityof FOV portions.
 35. A non-transitory computer-readable storage mediumfor high resolution time-of-flight depth imaging, the non-transitorycomputer-readable storage medium comprising: instructions stored thereinwhich, when executed by one or more processors, cause the one or moreprocessors to: illuminate, using one or more light-emitting devices of aprojection system, a plurality of field-of-view (FOV) portions of anentire FOV of the projection system, wherein each light-emitting deviceof the one or more light-emitting devices is configured to illuminate atleast one portion of the entire FOV; sequentially capture, by a sensorof a receiving system, a plurality of images based on a plurality ofillumination reflections corresponding to light emitted by the one ormore light-emitting devices, wherein each image of the plurality ofimages corresponds to one of the plurality of FOV portions, and whereinan image resolution associated with each image of the plurality ofimages corresponds to full a resolution of the sensor; and generate,using the plurality of images, an increased resolution depth mapassociated with the entire FOV.